Sep 08 13:16:10.965314 ip-10-0-137-66 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 13:16:11.383553 ip-10-0-137-66 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:11.383553 ip-10-0-137-66 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 13:16:11.383553 ip-10-0-137-66 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:11.383553 ip-10-0-137-66 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 13:16:11.383553 ip-10-0-137-66 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 13:16:11.386330 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.386232    2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 13:16:11.390263 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390244    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:11.390263 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390263    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390267    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390271    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390274    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390276    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390279    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390282    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390286    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390289    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390292    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390295    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390297    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390300    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390305    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390309    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390318    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390320    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390323    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390326    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390329    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:11.390329 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390332    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390335    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390338    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390341    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390343    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390346    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390348    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390351    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390354    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390356    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390359    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390361    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390364    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390366    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390369    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390371    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390374    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390376    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390379    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390382    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:11.390858 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390385    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390387    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390391    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390393    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390395    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390398    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390400    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390402    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390405    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390408    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390411    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390413    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390416    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390420    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390424    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390426    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390429    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390432    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390434    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:11.391350 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390437    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390439    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390442    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390444    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390447    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390449    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390452    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390454    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390456    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390459    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390461    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390464    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390466    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390469    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390472    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390475    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390477    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390482    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390486    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390488    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:11.392084 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390491    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390494    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390496    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390500    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390503    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390506    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390509    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390511    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390514    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390516    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390519    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390521    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390524    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390526    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390528    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390531    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390534    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390536    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:11.392649 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.390539    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392922    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392935    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392939    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392942    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392945    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392948    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392950    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392953    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392956    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392959    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392961    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392964    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392966    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392969    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392971    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392974    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392977    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392979    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392982    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:11.393119 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392985    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392987    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392990    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392992    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392995    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.392997    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393000    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393002    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393005    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393026    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393030    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393033    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393039    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393042    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393045    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393048    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393051    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393053    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393056    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393058    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:11.393606 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393060    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393063    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393065    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393068    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393070    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393073    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393075    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393078    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393080    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393082    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393085    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393088    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393090    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393093    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393096    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393099    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393103    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393107    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393109    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:11.394103 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393112    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393115    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393117    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393120    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393122    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393125    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393128    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393131    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393134    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393136    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393139    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393141    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393144    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393147    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393150    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393154    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393158    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393161    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393164    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393168    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:11.394574 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393172    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393176    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393180    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393183    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393186    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393190    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393193    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393195    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393198    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393200    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393203    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393206    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393208    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393210    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393213    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393216    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393218    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393222    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393226    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:11.395109 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393230    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393233    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393648    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393654    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393658    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393660    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393663    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393665    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393668    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393670    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393673    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393675    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393678    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393680    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393683    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393685    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393687    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393691    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393694    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393697    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:11.395562 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393699    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393702    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393705    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393707    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393709    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393712    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393715    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393717    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393720    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393722    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393725    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393727    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393730    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393732    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393736    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393740    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393743    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393746    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393748    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393751    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:11.396144 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393753    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393756    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393758    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393761    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393764    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393766    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393769    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393771    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393775    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393779    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393781    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393787    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393790    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393793    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393796    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393798    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393801    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393803    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393805    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:11.396656 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393808    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393810    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393813    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393815    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393817    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393820    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393822    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393825    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393828    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393830    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393833    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393835    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393837    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393840    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393842    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393845    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393847    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393849    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393852    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:11.397128 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393854    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393857    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393860    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393863    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393865    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393868    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393871    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393873    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393876    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393878    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393881    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393883    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393885    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393888    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393890    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393893    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393895    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393897    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393900    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393903    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:11.397578 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393905    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393907    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.393910    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.393983    2583 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394037    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394045    2583 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394050    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394059    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394062    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394066    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394071    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394074    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394077    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394080    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394084    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394087    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394089    2583 flags.go:64] FLAG: --cgroup-root=""
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394092    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394095    2583 flags.go:64] FLAG: --client-ca-file=""
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394099    2583 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394102    2583 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394106    2583 flags.go:64] FLAG: --cluster-domain=""
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394109    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 13:16:11.398105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394112    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394115    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394118    2583 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394125    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394128    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394131    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394134    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394137    2583 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394139    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394142    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394145    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394148    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394152    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394155    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394157    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394160    2583 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394163    2583 flags.go:64] FLAG: --enable-server="true"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394166    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394171    2583 flags.go:64] FLAG: --event-burst="100"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394174    2583 flags.go:64] FLAG: --event-qps="50"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394177    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394180    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394182    2583 flags.go:64] FLAG: --eviction-hard=""
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394187    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 13:16:11.398650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394189    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394192    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394195    2583 flags.go:64] FLAG: --eviction-soft=""
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394198    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394200    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394204    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394207    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394210    2583 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394212    2583 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394215    2583 flags.go:64] FLAG: --feature-gates=""
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394219    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394223    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394226    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394229    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394232    2583 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394235    2583 flags.go:64] FLAG: --help="false"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394238    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394241    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394243    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394246    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394250    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394253    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394256    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 13:16:11.399235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394258    2583 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394261    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394264    2583 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394267    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394269    2583 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394272    2583 flags.go:64] FLAG: --kube-reserved=""
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394275    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394278    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394280    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394283    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394286    2583 flags.go:64] FLAG: --lock-file=""
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394288    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394291    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394294    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394299    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394302    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394305    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394308    2583 flags.go:64] FLAG: --logging-format="text"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394310    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394314    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394317    2583 flags.go:64] FLAG: --manifest-url=""
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394320    2583 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394324    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394327    2583 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394332    2583 flags.go:64] FLAG: --max-pods="110"
Sep 08 13:16:11.399812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394335    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394338    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394340    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394343    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394346    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394349    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394351    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394359    2583 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394362    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394365    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394368    2583 flags.go:64] FLAG: --pod-cidr=""
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394371    2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394376    2583 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394379    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394382    2583 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394385    2583 flags.go:64] FLAG: --port="10250"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394388    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394391    2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01ee41bb137fc8412"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394394    2583 flags.go:64] FLAG: --qos-reserved=""
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394397    2583 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394399    2583 flags.go:64] FLAG: --register-node="true"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394402    2583 flags.go:64] FLAG: --register-with-taints=""
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394405    2583 flags.go:64] FLAG: --registry-burst="10"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394409    2583 flags.go:64] FLAG: --registry-qps="5"
Sep 08 13:16:11.400432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394411    2583 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394414    2583 flags.go:64] FLAG: --reserved-memory=""
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394417    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394420    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394424    2583 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394427    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394430    2583 flags.go:64] FLAG: --runonce="false"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394433    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394435    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394438    2583 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394441    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394444    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394447    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394450    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394454    2583 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394456    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394459    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394462    2583 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394465    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394468    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394471    2583 flags.go:64] FLAG: --system-cgroups=""
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394473    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394478    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394481    2583 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394484    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 13:16:11.401029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394488    2583 flags.go:64] FLAG: --tls-min-version=""
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394490    2583 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394493    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394496    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394499    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394501    2583 flags.go:64] FLAG: --v="2"
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394505    2583 flags.go:64] FLAG: --version="false"
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394510    2583 flags.go:64] FLAG: --vmodule=""
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394513    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.394516    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394648    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394653    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394656    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394661    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394664    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394666    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394669    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394672    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394675    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394678    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394680    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394683    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:11.401653 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394685    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394688    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394690    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394693    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394695    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394698    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394700    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394703    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394706    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394708    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394711    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394713    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394715    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394718    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394721    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394723    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394725    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394728    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394731    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:11.402319 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394733    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394737    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394742    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394745    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394750    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394752    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394755    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394757    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394760    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394762    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394764    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394767    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394769    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394784    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394788    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394790    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394793    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394797    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394800    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:11.403164 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394804    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394807    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394810    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394812    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394815    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394818    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394821    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394824    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394826    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394829    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394832    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394834    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394837    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394840    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394843    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394847    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394850    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394853    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394856    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394859    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:11.403970 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394861    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394864    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394866    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394870    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394873    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394876    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394879    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394881    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394884    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394886    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394889    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394891    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394894    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394896    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394899    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394901    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394904    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394906    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394909    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394911    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:11.404819 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394914    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394916    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394919    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394921    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394923    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394926    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394929    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394932    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.394935    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.395637    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.404363    2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.404385    2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.404413    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.404422    2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404504    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:11.405714 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404513    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404519    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404524    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404528    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404534    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404538    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404543    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404548    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404552    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404556    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404560    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404565    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404569    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404574    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404578    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404583    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404587    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404591    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404595    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404599    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:11.406239 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404606    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404611    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404616    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404621    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404626    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404630    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404635    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404639    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404645    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404651    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404657    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404661    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404665    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404670    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404674    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404678    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404682    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404686    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404690    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:11.406833 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404694    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404699    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404703    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404707    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404711    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404715    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404719    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404724    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404728    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404731    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404735    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404740    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404744    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404748    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404752    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404756    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404786    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404793    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404798    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404803    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:11.407313 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404808    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404813    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404818    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404823    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404827    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404831    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404835    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404839    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404844    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404848    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404853    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404857    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404861    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404866    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404870    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404874    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404878    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404882    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404886    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404890    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404894    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:11.407990 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404898    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404904    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404909    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404914    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404918    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404923    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404927    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404931    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404936    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404940    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404944    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404948    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404953    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404957    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404962    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404966    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404970    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:11.408871 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.404974    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.404982    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405190    2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405200    2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405205    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405209    2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405214    2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405218    2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405222    2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405226    2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405231    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405235    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405239    2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405243    2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405247    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 13:16:11.409335 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405251    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405255    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405259    2583 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405263    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405267    2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405271    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405275    2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405279    2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405283    2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405287    2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405291    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405295    2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405300    2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405304    2583 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405308    2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405319    2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405323    2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405328    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405331    2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 13:16:11.409809 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405335    2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405341    2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405347    2583 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405352    2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405356    2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405360    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405364    2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405368    2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405372    2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405376    2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405380    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405385    2583 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405389    2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405393    2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405397    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405401    2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405405    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405410    2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405414    2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405418    2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 13:16:11.410320 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405422    2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405426    2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405429    2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405433    2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405437    2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405442    2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405446    2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405450    2583 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405455    2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405460    2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405464    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405468    2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405472    2583 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405476    2583 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405480    2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405484    2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405488    2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405493    2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405497    2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405501    2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 13:16:11.410845 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405505    2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405509    2583 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405513    2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405517    2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405521    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405525    2583 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405529    2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405534    2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405538    2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405542    2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405546    2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405550    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405554    2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405558    2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405562    2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405566    2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405571    2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405576    2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405580    2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405584    2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 13:16:11.411374 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405588    2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405592    2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405597    2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405601    2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405607    2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405613    2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:11.405618    2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.405626    2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.405798    2583 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.409624    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.410802    2583 server.go:1019] "Starting client certificate rotation"
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.410904    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:11.411906 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.410948    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 13:16:11.414753 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.414735    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:11.415919 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.415897    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 13:16:11.437234 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.437215    2583 log.go:25] "Validated CRI v1 runtime API"
Sep 08 13:16:11.440753 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.440730    2583 log.go:25] "Validated CRI v1 image API"
Sep 08 13:16:11.441635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.441616    2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 13:16:11.445978 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.445957    2583 fs.go:135] Filesystem UUIDs: map[04d0f87f-bf76-4eb3-9abc-894e5622c6b9:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8e78ac64-7df7-40f2-bebd-c74c18d400e6:/dev/nvme0n1p4]
Sep 08 13:16:11.446077 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.445976    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 13:16:11.451760 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.451602    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:11.451852 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.451550    2583 manager.go:216] Machine: {Timestamp:2026-09-08 13:16:11.450104117 +0000 UTC m=+0.379007862 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083811 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec264a351006519c0a411f1194ec030d SystemUUID:ec264a35-1006-519c-0a41-1f1194ec030d BootID:a522d849-326c-4187-a77e-78ab585874e1 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f8:06:e9:fd:43 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f8:06:e9:fd:43 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:bb:31:6f:e2:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 13:16:11.451913 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.451853    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 13:16:11.451955 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.451943    2583 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 13:16:11.453752 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.453727    2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 13:16:11.453899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.453755    2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-66.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 13:16:11.455239 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.455229    2583 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 13:16:11.455273 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.455242    2583 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 13:16:11.455273 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.455250    2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:11.456047 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.456038    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 13:16:11.457916 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.457907    2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 13:16:11.461810 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.461799    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:11.461943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.461933    2583 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 13:16:11.461986 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.461966    2583 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 13:16:11.461986 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.461976    2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 13:16:11.462079 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.461991    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 13:16:11.462079 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.462001    2583 kubelet.go:398] "Adding apiserver pod source"
Sep 08 13:16:11.462079 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.462023    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 13:16:11.462375 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.462360    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:11.462423 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.462386    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 13:16:11.465537 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.465515    2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 13:16:11.468687 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.468672    2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 13:16:11.468757 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.468704    2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 13:16:11.469928 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469913    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 13:16:11.469992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469932    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 13:16:11.469992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469939    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 13:16:11.469992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469945    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 13:16:11.469992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469954    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 13:16:11.469992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469960    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 13:16:11.469992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469974    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 13:16:11.469992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469980    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 13:16:11.469992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469987    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 13:16:11.469992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.469993    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 13:16:11.470243 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.470001    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 13:16:11.470243 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.470020    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 13:16:11.470827 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.470810    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 13:16:11.470827 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.470821    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 13:16:11.474509 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.474463    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-66.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 13:16:11.474575 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.474516    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 13:16:11.474608 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.474560    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 13:16:11.474763 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.474751    2583 server.go:1268] "Started kubelet"
Sep 08 13:16:11.474856 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.474805    2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 13:16:11.474943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.474891    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 13:16:11.474995 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.474972    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 13:16:11.475772 ip-10-0-137-66 systemd[1]: Started Kubernetes Kubelet.
Sep 08 13:16:11.480233 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.480215    2583 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 13:16:11.481212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.481148    2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 13:16:11.482985 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.482964    2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 13:16:11.485548 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.484483    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-66.ec2.internal.18d35ada7ee80a5c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-66.ec2.internal,UID:ip-10-0-137-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-66.ec2.internal,},FirstTimestamp:2026-09-08 13:16:11.474725468 +0000 UTC m=+0.403629202,LastTimestamp:2026-09-08 13:16:11.474725468 +0000 UTC m=+0.403629202,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-66.ec2.internal,}"
Sep 08 13:16:11.487931 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.487914    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:11.488573 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.488551    2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 13:16:11.488683 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.488666    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:11.489232 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489217    2583 factory.go:55] Registering systemd factory
Sep 08 13:16:11.489317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489236    2583 factory.go:223] Registration of the systemd container factory successfully
Sep 08 13:16:11.489413 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489391    2583 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 13:16:11.489498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489418    2583 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 13:16:11.489498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489466    2583 factory.go:153] Registering CRI-O factory
Sep 08 13:16:11.489498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489480    2583 factory.go:223] Registration of the crio container factory successfully
Sep 08 13:16:11.489636 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489514    2583 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 13:16:11.489636 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489519    2583 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 13:16:11.489636 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489529    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 13:16:11.489636 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489377    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 13:16:11.489636 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489548    2583 factory.go:103] Registering Raw factory
Sep 08 13:16:11.489636 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489561    2583 manager.go:1196] Started watching for new ooms in manager
Sep 08 13:16:11.489941 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.489927    2583 manager.go:349] Starting recovery of all containers
Sep 08 13:16:11.498761 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.498747    2583 manager.go:354] Recovery completed
Sep 08 13:16:11.499202 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.499180    2583 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-66.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 13:16:11.499279 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.499211    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 13:16:11.503175 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.503161    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:11.506680 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.506657    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:11.506759 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.506696    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:11.506759 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.506710    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:11.507273 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.507262    2583 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 13:16:11.507273 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.507272    2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 13:16:11.507351 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.507278    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:11.507351 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.507287    2583 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 13:16:11.508161 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.508146    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:11.508206 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.508176    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:11.508206 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.508191    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:11.508439 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.508429    2583 policy_none.go:49] "None policy: Start"
Sep 08 13:16:11.508477 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.508445    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 13:16:11.508477 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.508455    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 13:16:11.509185 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.509118    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-66.ec2.internal.18d35ada80cf9ce1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-66.ec2.internal,UID:ip-10-0-137-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-66.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-66.ec2.internal,},FirstTimestamp:2026-09-08 13:16:11.506679009 +0000 UTC m=+0.435582752,LastTimestamp:2026-09-08 13:16:11.506679009 +0000 UTC m=+0.435582752,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-66.ec2.internal,}"
Sep 08 13:16:11.509481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.509472    2583 policy_none.go:47] "Start"
Sep 08 13:16:11.511336 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.511311    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4ctqp"
Sep 08 13:16:11.518284 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.518174    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-66.ec2.internal.18d35ada80cff6d2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-66.ec2.internal,UID:ip-10-0-137-66.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-66.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-66.ec2.internal,},FirstTimestamp:2026-09-08 13:16:11.506702034 +0000 UTC m=+0.435605768,LastTimestamp:2026-09-08 13:16:11.506702034 +0000 UTC m=+0.435605768,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-66.ec2.internal,}"
Sep 08 13:16:11.519674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.519655    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4ctqp"
Sep 08 13:16:11.545702 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.545686    2583 manager.go:337] "Starting Device Plugin manager"
Sep 08 13:16:11.561203 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.545733    2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 13:16:11.561203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.545744    2583 server.go:87] "Starting device plugin registration server"
Sep 08 13:16:11.561203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.546121    2583 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 13:16:11.561203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.546133    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 13:16:11.561203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.546244    2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 13:16:11.561203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.546348    2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 13:16:11.561203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.546358    2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 13:16:11.561203 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.546882    2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 13:16:11.561203 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.546931    2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:11.587492 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.587445    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 13:16:11.588943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.588924    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 13:16:11.589028 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.588994    2583 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 13:16:11.589086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.589042    2583 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 13:16:11.589135 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.589085    2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 13:16:11.591820 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.591800    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:11.646969 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.646857    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:11.647905 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.647888    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:11.647979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.647920    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:11.647979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.647930    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:11.647979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.647958    2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.657317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.657287    2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.657317 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.657323    2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-66.ec2.internal\": node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:11.689649 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.689620    2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal"]
Sep 08 13:16:11.689769 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.689710    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:11.689813 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.689796    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:11.691151 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.691135    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:11.691236 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.691164    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:11.691236 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.691175    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:11.692455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.692444    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:11.692595 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.692582    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.692631 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.692611    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:11.693241 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.693225    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:11.693317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.693248    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:11.693317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.693258    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:11.693317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.693273    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:11.693317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.693295    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:11.693317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.693311    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:11.694482 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.694466    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.694568 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.694491    2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 13:16:11.695213 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.695195    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 13:16:11.695298 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.695221    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 13:16:11.695298 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.695232    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeHasSufficientPID"
Sep 08 13:16:11.724756 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.724729    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-66.ec2.internal\" not found" node="ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.729096 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.729079    2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-66.ec2.internal\" not found" node="ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.790485 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.790454    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:11.890855 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.890824    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:11.890962 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.890876    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d7613509fc17d1105026a5ba25bea04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal\" (UID: \"4d7613509fc17d1105026a5ba25bea04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.890962 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.890922    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d7613509fc17d1105026a5ba25bea04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal\" (UID: \"4d7613509fc17d1105026a5ba25bea04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.891056 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.890966    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/727e83fa5ccafe51a7ba3765918be5a3-config\") pod \"kube-apiserver-proxy-ip-10-0-137-66.ec2.internal\" (UID: \"727e83fa5ccafe51a7ba3765918be5a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.991357 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:11.991280    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:11.991357 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.991326    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/727e83fa5ccafe51a7ba3765918be5a3-config\") pod \"kube-apiserver-proxy-ip-10-0-137-66.ec2.internal\" (UID: \"727e83fa5ccafe51a7ba3765918be5a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.991466 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.991367    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d7613509fc17d1105026a5ba25bea04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal\" (UID: \"4d7613509fc17d1105026a5ba25bea04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.991466 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.991392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d7613509fc17d1105026a5ba25bea04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal\" (UID: \"4d7613509fc17d1105026a5ba25bea04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.991466 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.991400    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/727e83fa5ccafe51a7ba3765918be5a3-config\") pod \"kube-apiserver-proxy-ip-10-0-137-66.ec2.internal\" (UID: \"727e83fa5ccafe51a7ba3765918be5a3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.991466 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.991439    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4d7613509fc17d1105026a5ba25bea04-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal\" (UID: \"4d7613509fc17d1105026a5ba25bea04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:11.991581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:11.991477    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d7613509fc17d1105026a5ba25bea04-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal\" (UID: \"4d7613509fc17d1105026a5ba25bea04\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:12.027479 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.027448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:12.031810 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.031496    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:12.092337 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:12.092297    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:12.192750 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:12.192712    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:12.293275 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:12.293177    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:12.393681 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:12.393638    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:12.404904 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.404885    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:12.409160 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.409144    2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:12.410620 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.410606    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 13:16:12.410749 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.410734    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:16:12.410793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.410731    2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 13:16:12.493742 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:12.493711    2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-66.ec2.internal\" not found"
Sep 08 13:16:12.521688 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.521621    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 13:11:12 +0000 UTC" deadline="2028-03-05 03:32:26.478465498 +0000 UTC"
Sep 08 13:16:12.521688 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.521663    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13046h16m13.95680684s"
Sep 08 13:16:12.577962 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.577928    2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:12.589602 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.589576    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:12.602710 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.602686    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:12.603711 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.603695    2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal"
Sep 08 13:16:12.612069 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.612048    2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 13:16:12.632913 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:12.632879    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727e83fa5ccafe51a7ba3765918be5a3.slice/crio-4cf9d0525ae3480aa415d44a9a86baa66190ed2f432d0f9c26425a17df86a062 WatchSource:0}: Error finding container 4cf9d0525ae3480aa415d44a9a86baa66190ed2f432d0f9c26425a17df86a062: Status 404 returned error can't find the container with id 4cf9d0525ae3480aa415d44a9a86baa66190ed2f432d0f9c26425a17df86a062
Sep 08 13:16:12.637596 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:12.637579    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:16:12.716178 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:12.716150    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7613509fc17d1105026a5ba25bea04.slice/crio-da425c1428ae7919a5750bb82a2b9d44e7127886aa8ac800d02b305481c7f834 WatchSource:0}: Error finding container da425c1428ae7919a5750bb82a2b9d44e7127886aa8ac800d02b305481c7f834: Status 404 returned error can't find the container with id da425c1428ae7919a5750bb82a2b9d44e7127886aa8ac800d02b305481c7f834
Sep 08 13:16:13.463481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.463217    2583 apiserver.go:52] "Watching apiserver"
Sep 08 13:16:13.479793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.479747    2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 13:16:13.481712 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.481676    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-78brt","kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ch66t","openshift-image-registry/node-ca-vrfh7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal","openshift-network-operator/iptables-alerter-98nf9","openshift-ovn-kubernetes/ovnkube-node-b2fbc","kube-system/global-pull-secret-syncer-jr9bb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj","openshift-dns/node-resolver-l9frn","openshift-multus/multus-additional-cni-plugins-mw5r7","openshift-multus/multus-szhm7","openshift-multus/network-metrics-daemon-82vzh","openshift-network-diagnostics/network-check-target-44vpt"]
Sep 08 13:16:13.483887 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.483765    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.486317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.486291    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.486643 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.486622    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 13:16:13.486737 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.486696    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-42wsg\""
Sep 08 13:16:13.487108 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.487083    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 13:16:13.487588 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.487573    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 13:16:13.487774 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.487757    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 13:16:13.487823 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.487792    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 13:16:13.487956 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.487937    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.488094 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.487960    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 13:16:13.488484 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.488465    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 13:16:13.491605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.489915    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 13:16:13.491605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.490153    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 13:16:13.491605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.490399    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 13:16:13.491605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.490654    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 13:16:13.491605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.490915    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gqfzt\""
Sep 08 13:16:13.491605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.491158    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-sqfzw\""
Sep 08 13:16:13.491605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.491338    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:13.491605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.491374    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 13:16:13.491605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.491585    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:13.492056 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.491673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.492361 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.492342    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:13.492449 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.492424    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:13.493552 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.493395    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.494272 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.494109    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 13:16:13.494272 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.494128    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 13:16:13.494272 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.494139    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qvngp\""
Sep 08 13:16:13.494524 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.494504    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 13:16:13.497455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.495769    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 13:16:13.497455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.496024    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 13:16:13.497455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.496256    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7v6xz\""
Sep 08 13:16:13.497455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.496417    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:16:13.497455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.496733    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.500335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.498374    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:13.500335 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.498440    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:13.500335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.498863    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 13:16:13.500335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.499699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v5j7v\""
Sep 08 13:16:13.500335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.499855    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 13:16:13.500335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500049    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 13:16:13.500680 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500546    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b546af0c-d8b5-4ce6-a934-52eeab3ef44d-iptables-alerter-script\") pod \"iptables-alerter-98nf9\" (UID: \"b546af0c-d8b5-4ce6-a934-52eeab3ef44d\") " pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.500680 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500576    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-systemd-units\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.500680 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500602    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-run-ovn\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.500680 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-log-socket\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.500680 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5dbb570-2303-4ba8-848c-b8ac08d60865-system-cni-dir\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.500934 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500689    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-sysconfig\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.500934 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500712    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-lib-modules\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.500934 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500739    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-var-lib-openvswitch\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.500934 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500764    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-systemd\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.500934 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-sys\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.500934 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500812    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:13.500934 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500837    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-run-systemd\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.500934 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500913    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-etc-openvswitch\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.500988    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5d710a10-f153-4cad-9c8f-2482d1a4ebef-ovnkube-script-lib\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501013    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5dbb570-2303-4ba8-848c-b8ac08d60865-cnibin\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501039    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qpsz\" (UniqueName: \"kubernetes.io/projected/a5dbb570-2303-4ba8-848c-b8ac08d60865-kube-api-access-6qpsz\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501064    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-modprobe-d\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501073    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501104    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-var-lib-kubelet\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.501130    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b546af0c-d8b5-4ce6-a934-52eeab3ef44d-host-slash\") pod \"iptables-alerter-98nf9\" (UID: \"b546af0c-d8b5-4ce6-a934-52eeab3ef44d\") " pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501184    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-node-log\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501210    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d710a10-f153-4cad-9c8f-2482d1a4ebef-ovn-node-metrics-cert\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501237    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5dbb570-2303-4ba8-848c-b8ac08d60865-cni-binary-copy\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501261    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5dbb570-2303-4ba8-848c-b8ac08d60865-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501286    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5dbb570-2303-4ba8-848c-b8ac08d60865-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501310    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5dbb570-2303-4ba8-848c-b8ac08d60865-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.501356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501345    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-run-openvswitch\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-cni-netd\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501390    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5d710a10-f153-4cad-9c8f-2482d1a4ebef-env-overrides\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501412    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-sysctl-d\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501435    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-kubelet\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-slash\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501498    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-run-netns\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501540    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-cni-bin\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501570    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501601    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5ww5\" (UniqueName: \"kubernetes.io/projected/5d710a10-f153-4cad-9c8f-2482d1a4ebef-kube-api-access-v5ww5\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-kubernetes\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501650    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/817efad1-a9dc-4e98-ae2d-396f97d0eda9-host\") pod \"node-ca-vrfh7\" (UID: \"817efad1-a9dc-4e98-ae2d-396f97d0eda9\") " pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr75w\" (UniqueName: \"kubernetes.io/projected/b546af0c-d8b5-4ce6-a934-52eeab3ef44d-kube-api-access-sr75w\") pod \"iptables-alerter-98nf9\" (UID: \"b546af0c-d8b5-4ce6-a934-52eeab3ef44d\") " pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501730    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5d710a10-f153-4cad-9c8f-2482d1a4ebef-ovnkube-config\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501753    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-run\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501775    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/817efad1-a9dc-4e98-ae2d-396f97d0eda9-serviceca\") pod \"node-ca-vrfh7\" (UID: \"817efad1-a9dc-4e98-ae2d-396f97d0eda9\") " pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.502078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501805    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfmxj\" (UniqueName: \"kubernetes.io/projected/817efad1-a9dc-4e98-ae2d-396f97d0eda9-kube-api-access-pfmxj\") pod \"node-ca-vrfh7\" (UID: \"817efad1-a9dc-4e98-ae2d-396f97d0eda9\") " pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.502691 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501825    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-sysctl-conf\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.502691 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501848    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-host\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.502691 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501867    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4b6680b-cc21-46ee-a842-48b689876d97-etc-tuned\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.502691 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501921    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4b6680b-cc21-46ee-a842-48b689876d97-tmp\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.502691 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.501937    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t2bk\" (UniqueName: \"kubernetes.io/projected/a4b6680b-cc21-46ee-a842-48b689876d97-kube-api-access-9t2bk\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.503690 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.503489    2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 13:16:13.503973 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.503950    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.507273 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.506390    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 13:16:13.507273 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.506526    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:13.507273 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.506682    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-892xm\""
Sep 08 13:16:13.507273 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.506915    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 13:16:13.508812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.508769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.510353 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.510331    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 13:16:13.510444 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.510374    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 13:16:13.510690 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.510674    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-n4t8t\""
Sep 08 13:16:13.511423 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.511255    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cttmh\""
Sep 08 13:16:13.512114 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.511382    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 13:16:13.529302 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.529279    2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jrj72"
Sep 08 13:16:13.537647 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.537627    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jrj72"
Sep 08 13:16:13.591653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.591621    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 13:16:13.594790 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.594731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal" event={"ID":"4d7613509fc17d1105026a5ba25bea04","Type":"ContainerStarted","Data":"da425c1428ae7919a5750bb82a2b9d44e7127886aa8ac800d02b305481c7f834"}
Sep 08 13:16:13.596296 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.596269    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal" event={"ID":"727e83fa5ccafe51a7ba3765918be5a3","Type":"ContainerStarted","Data":"4cf9d0525ae3480aa415d44a9a86baa66190ed2f432d0f9c26425a17df86a062"}
Sep 08 13:16:13.602237 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-lib-modules\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.602345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602254    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/92745d4d-0c99-4d1b-a91e-b7a546e0e583-agent-certs\") pod \"konnectivity-agent-78brt\" (UID: \"92745d4d-0c99-4d1b-a91e-b7a546e0e583\") " pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:13.602345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-var-lib-openvswitch\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.602345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-sys\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.602345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/92745d4d-0c99-4d1b-a91e-b7a546e0e583-konnectivity-ca\") pod \"konnectivity-agent-78brt\" (UID: \"92745d4d-0c99-4d1b-a91e-b7a546e0e583\") " pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602396    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-var-lib-openvswitch\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602416    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-sys\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602423    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-var-lib-cni-bin\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602460    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5d710a10-f153-4cad-9c8f-2482d1a4ebef-ovnkube-script-lib\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-var-lib-kubelet\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602484    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-lib-modules\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602513    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt8sg\" (UniqueName: \"kubernetes.io/projected/02703e20-c856-4652-9e02-6086e7b11179-kube-api-access-mt8sg\") pod \"node-resolver-l9frn\" (UID: \"02703e20-c856-4652-9e02-6086e7b11179\") " pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602539    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-run-netns\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.602570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-conf-dir\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-var-lib-kubelet\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602588    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602619    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-socket-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602684    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b546af0c-d8b5-4ce6-a934-52eeab3ef44d-host-slash\") pod \"iptables-alerter-98nf9\" (UID: \"b546af0c-d8b5-4ce6-a934-52eeab3ef44d\") " pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-node-log\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602729    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d710a10-f153-4cad-9c8f-2482d1a4ebef-ovn-node-metrics-cert\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602744    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5dbb570-2303-4ba8-848c-b8ac08d60865-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602746    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b546af0c-d8b5-4ce6-a934-52eeab3ef44d-host-slash\") pod \"iptables-alerter-98nf9\" (UID: \"b546af0c-d8b5-4ce6-a934-52eeab3ef44d\") " pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjwp\" (UniqueName: \"kubernetes.io/projected/a1afd7b4-babf-47e5-b128-300b8dde301f-kube-api-access-lnjwp\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602813    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-node-log\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602822    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-cni-netd\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5d710a10-f153-4cad-9c8f-2482d1a4ebef-env-overrides\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602877    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nghtw\" (UniqueName: \"kubernetes.io/projected/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-kube-api-access-nghtw\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.602994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-var-lib-cni-multus\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603023    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-var-lib-kubelet\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.603235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603054    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-daemon-config\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603121    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5ww5\" (UniqueName: \"kubernetes.io/projected/5d710a10-f153-4cad-9c8f-2482d1a4ebef-kube-api-access-v5ww5\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-cni-dir\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603172    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr75w\" (UniqueName: \"kubernetes.io/projected/b546af0c-d8b5-4ce6-a934-52eeab3ef44d-kube-api-access-sr75w\") pod \"iptables-alerter-98nf9\" (UID: \"b546af0c-d8b5-4ce6-a934-52eeab3ef44d\") " pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603215    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfmxj\" (UniqueName: \"kubernetes.io/projected/817efad1-a9dc-4e98-ae2d-396f97d0eda9-kube-api-access-pfmxj\") pod \"node-ca-vrfh7\" (UID: \"817efad1-a9dc-4e98-ae2d-396f97d0eda9\") " pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603243    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-registration-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603269    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603293    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-run-multus-certs\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603321    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4b6680b-cc21-46ee-a842-48b689876d97-etc-tuned\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4b6680b-cc21-46ee-a842-48b689876d97-tmp\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603368    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02703e20-c856-4652-9e02-6086e7b11179-tmp-dir\") pod \"node-resolver-l9frn\" (UID: \"02703e20-c856-4652-9e02-6086e7b11179\") " pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b546af0c-d8b5-4ce6-a934-52eeab3ef44d-iptables-alerter-script\") pod \"iptables-alerter-98nf9\" (UID: \"b546af0c-d8b5-4ce6-a934-52eeab3ef44d\") " pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603416    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a5dbb570-2303-4ba8-848c-b8ac08d60865-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603497    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-sysconfig\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603541    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-cni-netd\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603419    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-sysconfig\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603580    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-sys-fs\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.603899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603607    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603632    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-cnibin\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603657    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-run-k8s-cni-cncf-io\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-systemd\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/817efad1-a9dc-4e98-ae2d-396f97d0eda9-host\") pod \"node-ca-vrfh7\" (UID: \"817efad1-a9dc-4e98-ae2d-396f97d0eda9\") " pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-run-systemd\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5d710a10-f153-4cad-9c8f-2482d1a4ebef-ovnkube-script-lib\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603764    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-etc-openvswitch\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603790    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603816    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5dbb570-2303-4ba8-848c-b8ac08d60865-cnibin\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603844    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qpsz\" (UniqueName: \"kubernetes.io/projected/a5dbb570-2303-4ba8-848c-b8ac08d60865-kube-api-access-6qpsz\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-modprobe-d\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02703e20-c856-4652-9e02-6086e7b11179-hosts-file\") pod \"node-resolver-l9frn\" (UID: \"02703e20-c856-4652-9e02-6086e7b11179\") " pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603893    2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603937    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-sysctl-d\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5d710a10-f153-4cad-9c8f-2482d1a4ebef-env-overrides\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603969    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5dbb570-2303-4ba8-848c-b8ac08d60865-cni-binary-copy\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.604660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.603989    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5dbb570-2303-4ba8-848c-b8ac08d60865-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5dbb570-2303-4ba8-848c-b8ac08d60865-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604029    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-run-openvswitch\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-system-cni-dir\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604321    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-sysctl-d\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604365    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5dbb570-2303-4ba8-848c-b8ac08d60865-cnibin\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-run-systemd\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604405    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-etc-openvswitch\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-systemd\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/817efad1-a9dc-4e98-ae2d-396f97d0eda9-host\") pod \"node-ca-vrfh7\" (UID: \"817efad1-a9dc-4e98-ae2d-396f97d0eda9\") " pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604539    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5dbb570-2303-4ba8-848c-b8ac08d60865-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604733    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5dbb570-2303-4ba8-848c-b8ac08d60865-cni-binary-copy\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604782    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-run-openvswitch\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-kubelet\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604845    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-run-ovn-kubernetes\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604848    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-slash\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604912    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-run-netns\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.605405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604918    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-modprobe-d\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-cni-bin\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.604973    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605006    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-slash\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605012    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-kubernetes\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605014    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a5dbb570-2303-4ba8-848c-b8ac08d60865-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605049    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-kubelet\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605052    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-dbus\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605072    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605105    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b546af0c-d8b5-4ce6-a934-52eeab3ef44d-iptables-alerter-script\") pod \"iptables-alerter-98nf9\" (UID: \"b546af0c-d8b5-4ce6-a934-52eeab3ef44d\") " pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605120    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-socket-dir-parent\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-run-netns\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-kubernetes\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605179    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5d710a10-f153-4cad-9c8f-2482d1a4ebef-ovnkube-config\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605223    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-run\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605224    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-host-cni-bin\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605299    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/817efad1-a9dc-4e98-ae2d-396f97d0eda9-serviceca\") pod \"node-ca-vrfh7\" (UID: \"817efad1-a9dc-4e98-ae2d-396f97d0eda9\") " pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.606269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d533862a-b3ec-4f39-9018-e357ba3f02c5-cni-binary-copy\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605413    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-run\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605430    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-etc-kubernetes\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605467    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp86b\" (UniqueName: \"kubernetes.io/projected/d533862a-b3ec-4f39-9018-e357ba3f02c5-kube-api-access-bp86b\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-sysctl-conf\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605549    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-host\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605594    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-host\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t2bk\" (UniqueName: \"kubernetes.io/projected/a4b6680b-cc21-46ee-a842-48b689876d97-kube-api-access-9t2bk\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605723    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5d710a10-f153-4cad-9c8f-2482d1a4ebef-ovnkube-config\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605729    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/817efad1-a9dc-4e98-ae2d-396f97d0eda9-serviceca\") pod \"node-ca-vrfh7\" (UID: \"817efad1-a9dc-4e98-ae2d-396f97d0eda9\") " pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-device-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-kubelet-config\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605839    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4b6680b-cc21-46ee-a842-48b689876d97-etc-sysctl-conf\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605852    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-hostroot\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605884    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-systemd-units\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605939    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-run-ovn\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-log-socket\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.607133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605971    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-run-ovn\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.607973 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.605980    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5dbb570-2303-4ba8-848c-b8ac08d60865-system-cni-dir\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.607973 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.606046    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5dbb570-2303-4ba8-848c-b8ac08d60865-system-cni-dir\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.607973 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.606048    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-systemd-units\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.607973 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.606142    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5d710a10-f153-4cad-9c8f-2482d1a4ebef-log-socket\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.608778 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.608754    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:13.608874 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.608786    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:13.608874 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.608827    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rq4kd for pod openshift-network-diagnostics/network-check-target-44vpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:13.608985 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.608909    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd podName:374c967a-24a3-40b4-872f-bfa954850357 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:14.108887011 +0000 UTC m=+3.037790770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rq4kd" (UniqueName: "kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd") pod "network-check-target-44vpt" (UID: "374c967a-24a3-40b4-872f-bfa954850357") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:13.611317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.610495    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4b6680b-cc21-46ee-a842-48b689876d97-tmp\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.611317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.610612    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4b6680b-cc21-46ee-a842-48b689876d97-etc-tuned\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.611317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.611277    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d710a10-f153-4cad-9c8f-2482d1a4ebef-ovn-node-metrics-cert\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.613351 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.613324    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfmxj\" (UniqueName: \"kubernetes.io/projected/817efad1-a9dc-4e98-ae2d-396f97d0eda9-kube-api-access-pfmxj\") pod \"node-ca-vrfh7\" (UID: \"817efad1-a9dc-4e98-ae2d-396f97d0eda9\") " pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.613723 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.613700    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5ww5\" (UniqueName: \"kubernetes.io/projected/5d710a10-f153-4cad-9c8f-2482d1a4ebef-kube-api-access-v5ww5\") pod \"ovnkube-node-b2fbc\" (UID: \"5d710a10-f153-4cad-9c8f-2482d1a4ebef\") " pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.614712 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.614689    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr75w\" (UniqueName: \"kubernetes.io/projected/b546af0c-d8b5-4ce6-a934-52eeab3ef44d-kube-api-access-sr75w\") pod \"iptables-alerter-98nf9\" (UID: \"b546af0c-d8b5-4ce6-a934-52eeab3ef44d\") " pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.615481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.615094    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t2bk\" (UniqueName: \"kubernetes.io/projected/a4b6680b-cc21-46ee-a842-48b689876d97-kube-api-access-9t2bk\") pod \"tuned-ch66t\" (UID: \"a4b6680b-cc21-46ee-a842-48b689876d97\") " pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.615481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.615447    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qpsz\" (UniqueName: \"kubernetes.io/projected/a5dbb570-2303-4ba8-848c-b8ac08d60865-kube-api-access-6qpsz\") pod \"multus-additional-cni-plugins-mw5r7\" (UID: \"a5dbb570-2303-4ba8-848c-b8ac08d60865\") " pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.707073 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707018    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02703e20-c856-4652-9e02-6086e7b11179-hosts-file\") pod \"node-resolver-l9frn\" (UID: \"02703e20-c856-4652-9e02-6086e7b11179\") " pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.707073 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707086    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-system-cni-dir\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-dbus\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:13.707335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02703e20-c856-4652-9e02-6086e7b11179-hosts-file\") pod \"node-resolver-l9frn\" (UID: \"02703e20-c856-4652-9e02-6086e7b11179\") " pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.707335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-socket-dir-parent\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707252    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d533862a-b3ec-4f39-9018-e357ba3f02c5-cni-binary-copy\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-dbus\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:13.707335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707275    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-etc-kubernetes\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp86b\" (UniqueName: \"kubernetes.io/projected/d533862a-b3ec-4f39-9018-e357ba3f02c5-kube-api-access-bp86b\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707346    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-device-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707373    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-kubelet-config\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-hostroot\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707431    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/92745d4d-0c99-4d1b-a91e-b7a546e0e583-agent-certs\") pod \"konnectivity-agent-78brt\" (UID: \"92745d4d-0c99-4d1b-a91e-b7a546e0e583\") " pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707475    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/92745d4d-0c99-4d1b-a91e-b7a546e0e583-konnectivity-ca\") pod \"konnectivity-agent-78brt\" (UID: \"92745d4d-0c99-4d1b-a91e-b7a546e0e583\") " pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707501    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-var-lib-cni-bin\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707187    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-system-cni-dir\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707527    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt8sg\" (UniqueName: \"kubernetes.io/projected/02703e20-c856-4652-9e02-6086e7b11179-kube-api-access-mt8sg\") pod \"node-resolver-l9frn\" (UID: \"02703e20-c856-4652-9e02-6086e7b11179\") " pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707581    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-run-netns\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707609    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-conf-dir\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707664    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-socket-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.707716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjwp\" (UniqueName: \"kubernetes.io/projected/a1afd7b4-babf-47e5-b128-300b8dde301f-kube-api-access-lnjwp\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707752    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nghtw\" (UniqueName: \"kubernetes.io/projected/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-kube-api-access-nghtw\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707766    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-hostroot\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707781    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-var-lib-cni-multus\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707808    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-var-lib-kubelet\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707834    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-daemon-config\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707862    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-cni-dir\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707892    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-registration-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707910    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d533862a-b3ec-4f39-9018-e357ba3f02c5-cni-binary-copy\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707315    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-etc-kubernetes\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707986    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-device-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707267    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-socket-dir-parent\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.708019    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708043    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-kubelet-config\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708108    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-run-netns\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.708115    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret podName:de8493b4-7db2-438d-bcaa-b97e00b8c2e3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:14.208072073 +0000 UTC m=+3.136975824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret") pod "global-pull-secret-syncer-jr9bb" (UID: "de8493b4-7db2-438d-bcaa-b97e00b8c2e3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708152    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-var-lib-cni-multus\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.708481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708200    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-var-lib-kubelet\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.707918    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708310    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-conf-dir\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708333    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-run-multus-certs\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708355    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02703e20-c856-4652-9e02-6086e7b11179-tmp-dir\") pod \"node-resolver-l9frn\" (UID: \"02703e20-c856-4652-9e02-6086e7b11179\") " pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708385    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708407    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-sys-fs\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708521    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-socket-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708611    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-daemon-config\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708635    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-etc-selinux\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708694    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-multus-cni-dir\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708721    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/92745d4d-0c99-4d1b-a91e-b7a546e0e583-konnectivity-ca\") pod \"konnectivity-agent-78brt\" (UID: \"92745d4d-0c99-4d1b-a91e-b7a546e0e583\") " pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708747    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-registration-dir\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708794    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-run-multus-certs\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708849    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-var-lib-cni-bin\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a1afd7b4-babf-47e5-b128-300b8dde301f-sys-fs\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708942    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.708979    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-cnibin\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.709005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02703e20-c856-4652-9e02-6086e7b11179-tmp-dir\") pod \"node-resolver-l9frn\" (UID: \"02703e20-c856-4652-9e02-6086e7b11179\") " pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.709855 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.709007    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-run-k8s-cni-cncf-io\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709855 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.709046    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:13.709855 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.709089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-cnibin\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.709855 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:13.709107    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs podName:29de17e1-0cb6-41fb-bd61-0a1fc39b19b3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:14.209086575 +0000 UTC m=+3.137990301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs") pod "network-metrics-daemon-82vzh" (UID: "29de17e1-0cb6-41fb-bd61-0a1fc39b19b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:13.710297 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.709129    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d533862a-b3ec-4f39-9018-e357ba3f02c5-host-run-k8s-cni-cncf-io\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.713221 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.711643    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/92745d4d-0c99-4d1b-a91e-b7a546e0e583-agent-certs\") pod \"konnectivity-agent-78brt\" (UID: \"92745d4d-0c99-4d1b-a91e-b7a546e0e583\") " pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:13.717735 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.717658    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nghtw\" (UniqueName: \"kubernetes.io/projected/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-kube-api-access-nghtw\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:13.718804 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.718776    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp86b\" (UniqueName: \"kubernetes.io/projected/d533862a-b3ec-4f39-9018-e357ba3f02c5-kube-api-access-bp86b\") pod \"multus-szhm7\" (UID: \"d533862a-b3ec-4f39-9018-e357ba3f02c5\") " pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.719298 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.719273    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt8sg\" (UniqueName: \"kubernetes.io/projected/02703e20-c856-4652-9e02-6086e7b11179-kube-api-access-mt8sg\") pod \"node-resolver-l9frn\" (UID: \"02703e20-c856-4652-9e02-6086e7b11179\") " pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.721842 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.721821    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjwp\" (UniqueName: \"kubernetes.io/projected/a1afd7b4-babf-47e5-b128-300b8dde301f-kube-api-access-lnjwp\") pod \"aws-ebs-csi-driver-node-zftxj\" (UID: \"a1afd7b4-babf-47e5-b128-300b8dde301f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.802936 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.802894    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:13.811194 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.811170    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mw5r7"
Sep 08 13:16:13.828429 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.828403    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ch66t"
Sep 08 13:16:13.830845 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.830815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vrfh7"
Sep 08 13:16:13.837453 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.837429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-98nf9"
Sep 08 13:16:13.848216 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.848189    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj"
Sep 08 13:16:13.857056 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.857040    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l9frn"
Sep 08 13:16:13.864888 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.864872    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:13.871628 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.871597    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-szhm7"
Sep 08 13:16:13.920939 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:13.920908    2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:14.112438 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:14.112396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:14.112635 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:14.112570    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:14.112635 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:14.112595    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:14.112635 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:14.112608    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rq4kd for pod openshift-network-diagnostics/network-check-target-44vpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:14.112811 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:14.112673    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd podName:374c967a-24a3-40b4-872f-bfa954850357 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:15.112654362 +0000 UTC m=+4.041558103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rq4kd" (UniqueName: "kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd") pod "network-check-target-44vpt" (UID: "374c967a-24a3-40b4-872f-bfa954850357") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:14.213170 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:14.213128    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:14.213351 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:14.213191    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:14.213351 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:14.213261    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:14.213351 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:14.213333    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret podName:de8493b4-7db2-438d-bcaa-b97e00b8c2e3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:15.213313172 +0000 UTC m=+4.142216912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret") pod "global-pull-secret-syncer-jr9bb" (UID: "de8493b4-7db2-438d-bcaa-b97e00b8c2e3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:14.213517 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:14.213368    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:14.213517 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:14.213450    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs podName:29de17e1-0cb6-41fb-bd61-0a1fc39b19b3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:15.213430061 +0000 UTC m=+4.142333789 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs") pod "network-metrics-daemon-82vzh" (UID: "29de17e1-0cb6-41fb-bd61-0a1fc39b19b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:14.540818 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:14.540779    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:14 +0000 UTC" deadline="2028-05-22 02:06:15.714190206 +0000 UTC"
Sep 08 13:16:14.540818 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:14.540813    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14916h50m1.173380929s"
Sep 08 13:16:14.559563 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:14.559532    2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 13:16:14.589607 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:14.589566    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:14.589777 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:14.589734    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:14.871133 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:14.871103    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d710a10_f153_4cad_9c8f_2482d1a4ebef.slice/crio-4bad30fe6a6f8dbf312a08a81e74a847d85b07f98a159bc9c2ae6695a2c8f3e6 WatchSource:0}: Error finding container 4bad30fe6a6f8dbf312a08a81e74a847d85b07f98a159bc9c2ae6695a2c8f3e6: Status 404 returned error can't find the container with id 4bad30fe6a6f8dbf312a08a81e74a847d85b07f98a159bc9c2ae6695a2c8f3e6
Sep 08 13:16:14.904047 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:14.904016    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b6680b_cc21_46ee_a842_48b689876d97.slice/crio-1dde02a7501a5cbb7d4ced7a94e70e3733aa31a2958ea44011666c0f86f824e9 WatchSource:0}: Error finding container 1dde02a7501a5cbb7d4ced7a94e70e3733aa31a2958ea44011666c0f86f824e9: Status 404 returned error can't find the container with id 1dde02a7501a5cbb7d4ced7a94e70e3733aa31a2958ea44011666c0f86f824e9
Sep 08 13:16:14.905435 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:14.905397    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02703e20_c856_4652_9e02_6086e7b11179.slice/crio-6cd18a190a451b707fb0637d038859ad0585640e676556bd63f91b98c30a8881 WatchSource:0}: Error finding container 6cd18a190a451b707fb0637d038859ad0585640e676556bd63f91b98c30a8881: Status 404 returned error can't find the container with id 6cd18a190a451b707fb0637d038859ad0585640e676556bd63f91b98c30a8881
Sep 08 13:16:14.907192 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:14.907170    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb546af0c_d8b5_4ce6_a934_52eeab3ef44d.slice/crio-e58cd53d9dfe0fa8c8f3f554f50295bb5a0bb2132646f6b4d4a5153ee20a7a2d WatchSource:0}: Error finding container e58cd53d9dfe0fa8c8f3f554f50295bb5a0bb2132646f6b4d4a5153ee20a7a2d: Status 404 returned error can't find the container with id e58cd53d9dfe0fa8c8f3f554f50295bb5a0bb2132646f6b4d4a5153ee20a7a2d
Sep 08 13:16:14.908183 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:14.908164    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5dbb570_2303_4ba8_848c_b8ac08d60865.slice/crio-8319c50f45ac23450dc215a1d056bee09e07f6119ee62863b56e708ae62e1a92 WatchSource:0}: Error finding container 8319c50f45ac23450dc215a1d056bee09e07f6119ee62863b56e708ae62e1a92: Status 404 returned error can't find the container with id 8319c50f45ac23450dc215a1d056bee09e07f6119ee62863b56e708ae62e1a92
Sep 08 13:16:14.910912 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:14.910888    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd533862a_b3ec_4f39_9018_e357ba3f02c5.slice/crio-36646e2409664ca3ed715f33bd61ff8373a87249ab5f197f3ba6602e5094ed4e WatchSource:0}: Error finding container 36646e2409664ca3ed715f33bd61ff8373a87249ab5f197f3ba6602e5094ed4e: Status 404 returned error can't find the container with id 36646e2409664ca3ed715f33bd61ff8373a87249ab5f197f3ba6602e5094ed4e
Sep 08 13:16:14.913102 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:14.912855    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817efad1_a9dc_4e98_ae2d_396f97d0eda9.slice/crio-be25dffc4e84d46211a929d98b5886217ff27146be87fb9395f977840469906e WatchSource:0}: Error finding container be25dffc4e84d46211a929d98b5886217ff27146be87fb9395f977840469906e: Status 404 returned error can't find the container with id be25dffc4e84d46211a929d98b5886217ff27146be87fb9395f977840469906e
Sep 08 13:16:14.917091 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:14.916966    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92745d4d_0c99_4d1b_a91e_b7a546e0e583.slice/crio-bce37f90441ace8218e5b40dec3fa5f8fca51832feb6d34ca86330d9973b4291 WatchSource:0}: Error finding container bce37f90441ace8218e5b40dec3fa5f8fca51832feb6d34ca86330d9973b4291: Status 404 returned error can't find the container with id bce37f90441ace8218e5b40dec3fa5f8fca51832feb6d34ca86330d9973b4291
Sep 08 13:16:15.120270 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.120176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:15.120415 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.120326    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:15.120415 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.120342    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:15.120415 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.120351    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rq4kd for pod openshift-network-diagnostics/network-check-target-44vpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:15.120415 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.120407    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd podName:374c967a-24a3-40b4-872f-bfa954850357 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:17.120394182 +0000 UTC m=+6.049297904 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rq4kd" (UniqueName: "kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd") pod "network-check-target-44vpt" (UID: "374c967a-24a3-40b4-872f-bfa954850357") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:15.221403 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.221376    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:15.221550 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.221422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:15.221616 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.221565    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:15.221616 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.221577    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:15.221717 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.221637    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret podName:de8493b4-7db2-438d-bcaa-b97e00b8c2e3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:17.221616152 +0000 UTC m=+6.150519874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret") pod "global-pull-secret-syncer-jr9bb" (UID: "de8493b4-7db2-438d-bcaa-b97e00b8c2e3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:15.221717 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.221662    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs podName:29de17e1-0cb6-41fb-bd61-0a1fc39b19b3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:17.221652043 +0000 UTC m=+6.150555766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs") pod "network-metrics-daemon-82vzh" (UID: "29de17e1-0cb6-41fb-bd61-0a1fc39b19b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:15.541625 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.541488    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 13:11:14 +0000 UTC" deadline="2028-05-01 03:23:11.651957918 +0000 UTC"
Sep 08 13:16:15.541625 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.541533    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14414h6m56.110429729s"
Sep 08 13:16:15.590198 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.590162    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:15.590644 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.590295    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:15.592908 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.592882    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:15.593026 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:15.592997    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:15.603170 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.603135    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrfh7" event={"ID":"817efad1-a9dc-4e98-ae2d-396f97d0eda9","Type":"ContainerStarted","Data":"be25dffc4e84d46211a929d98b5886217ff27146be87fb9395f977840469906e"}
Sep 08 13:16:15.605684 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.605612    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-szhm7" event={"ID":"d533862a-b3ec-4f39-9018-e357ba3f02c5","Type":"ContainerStarted","Data":"36646e2409664ca3ed715f33bd61ff8373a87249ab5f197f3ba6602e5094ed4e"}
Sep 08 13:16:15.608169 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.608097    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerStarted","Data":"8319c50f45ac23450dc215a1d056bee09e07f6119ee62863b56e708ae62e1a92"}
Sep 08 13:16:15.611302 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.610986    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-98nf9" event={"ID":"b546af0c-d8b5-4ce6-a934-52eeab3ef44d","Type":"ContainerStarted","Data":"e58cd53d9dfe0fa8c8f3f554f50295bb5a0bb2132646f6b4d4a5153ee20a7a2d"}
Sep 08 13:16:15.614854 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.614698    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" event={"ID":"5d710a10-f153-4cad-9c8f-2482d1a4ebef","Type":"ContainerStarted","Data":"4bad30fe6a6f8dbf312a08a81e74a847d85b07f98a159bc9c2ae6695a2c8f3e6"}
Sep 08 13:16:15.622162 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.621652    2583 generic.go:356] "Generic (PLEG): container finished" podID="4d7613509fc17d1105026a5ba25bea04" containerID="3a148bbe1847e506609b88200e88021d2feddf4cdefe0614381137552a270ed8" exitCode=0
Sep 08 13:16:15.622162 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.621894    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal" event={"ID":"4d7613509fc17d1105026a5ba25bea04","Type":"ContainerDied","Data":"3a148bbe1847e506609b88200e88021d2feddf4cdefe0614381137552a270ed8"}
Sep 08 13:16:15.629502 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.629456    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal" event={"ID":"727e83fa5ccafe51a7ba3765918be5a3","Type":"ContainerStarted","Data":"47749461e724566b4490ba7bddcf1c56514e8ce8e59b31020baf92709ecb8fd5"}
Sep 08 13:16:15.633997 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.633966    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9frn" event={"ID":"02703e20-c856-4652-9e02-6086e7b11179","Type":"ContainerStarted","Data":"6cd18a190a451b707fb0637d038859ad0585640e676556bd63f91b98c30a8881"}
Sep 08 13:16:15.637812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.637781    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ch66t" event={"ID":"a4b6680b-cc21-46ee-a842-48b689876d97","Type":"ContainerStarted","Data":"1dde02a7501a5cbb7d4ced7a94e70e3733aa31a2958ea44011666c0f86f824e9"}
Sep 08 13:16:15.644844 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.644813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-78brt" event={"ID":"92745d4d-0c99-4d1b-a91e-b7a546e0e583","Type":"ContainerStarted","Data":"bce37f90441ace8218e5b40dec3fa5f8fca51832feb6d34ca86330d9973b4291"}
Sep 08 13:16:15.647357 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:15.647329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj" event={"ID":"a1afd7b4-babf-47e5-b128-300b8dde301f","Type":"ContainerStarted","Data":"57338e315010f7f9eb925845b60c0737ae246da0ef7da7fb2f1567edc3b94ff6"}
Sep 08 13:16:16.589651 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:16.589616    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:16.590207 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:16.589776    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:16.656662 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:16.656617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal" event={"ID":"4d7613509fc17d1105026a5ba25bea04","Type":"ContainerStarted","Data":"770f6a579c06e7b4f8dec20276f747f30199914c144723419ff9146b72d7997b"}
Sep 08 13:16:16.674442 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:16.673328    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-66.ec2.internal" podStartSLOduration=3.673289707 podStartE2EDuration="3.673289707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:15.658539189 +0000 UTC m=+4.587442936" watchObservedRunningTime="2026-09-08 13:16:16.673289707 +0000 UTC m=+5.602193442"
Sep 08 13:16:17.146217 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:17.145909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:17.146454 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.146438    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:17.146549 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.146462    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:17.146549 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.146476    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rq4kd for pod openshift-network-diagnostics/network-check-target-44vpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:17.146549 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.146543    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd podName:374c967a-24a3-40b4-872f-bfa954850357 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:21.146521978 +0000 UTC m=+10.075425735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rq4kd" (UniqueName: "kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd") pod "network-check-target-44vpt" (UID: "374c967a-24a3-40b4-872f-bfa954850357") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:17.247234 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:17.247185    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:17.247419 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:17.247256    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:17.247510 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.247489    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:17.247579 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.247565    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs podName:29de17e1-0cb6-41fb-bd61-0a1fc39b19b3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:21.247544902 +0000 UTC m=+10.176448625 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs") pod "network-metrics-daemon-82vzh" (UID: "29de17e1-0cb6-41fb-bd61-0a1fc39b19b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:17.248035 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.248011    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:17.248142 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.248080    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret podName:de8493b4-7db2-438d-bcaa-b97e00b8c2e3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:21.24806376 +0000 UTC m=+10.176967482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret") pod "global-pull-secret-syncer-jr9bb" (UID: "de8493b4-7db2-438d-bcaa-b97e00b8c2e3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:17.592861 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:17.592822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:17.593418 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.592977    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:17.593493 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:17.593429    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:17.593552 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:17.593527    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:18.590083 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:18.590048    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:18.590302 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:18.590212    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:19.589945 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:19.589909    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:19.590401 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:19.590076    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:19.590538 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:19.590519    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:19.590679 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:19.590658    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:20.590042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:20.589827    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:20.590042 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:20.590002    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:21.178954 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:21.178317    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:21.178954 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.178481    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:21.178954 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.178501    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:21.178954 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.178516    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rq4kd for pod openshift-network-diagnostics/network-check-target-44vpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:21.178954 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.178578    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd podName:374c967a-24a3-40b4-872f-bfa954850357 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:29.178559533 +0000 UTC m=+18.107463260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rq4kd" (UniqueName: "kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd") pod "network-check-target-44vpt" (UID: "374c967a-24a3-40b4-872f-bfa954850357") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:21.279829 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:21.279564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:21.279829 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:21.279657    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:21.279829 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.279742    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:21.279829 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.279740    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:21.279829 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.279799    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret podName:de8493b4-7db2-438d-bcaa-b97e00b8c2e3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:29.279782027 +0000 UTC m=+18.208685764 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret") pod "global-pull-secret-syncer-jr9bb" (UID: "de8493b4-7db2-438d-bcaa-b97e00b8c2e3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:21.279829 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.279814    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs podName:29de17e1-0cb6-41fb-bd61-0a1fc39b19b3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:29.2798071 +0000 UTC m=+18.208710820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs") pod "network-metrics-daemon-82vzh" (UID: "29de17e1-0cb6-41fb-bd61-0a1fc39b19b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:21.593621 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:21.593559    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:21.594034 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.593686    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:21.594034 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:21.593739    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:21.594034 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:21.593854    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:22.589337 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:22.589281    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:22.589529 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:22.589438    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:23.593975 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:23.593947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:23.594429 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:23.594097    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:23.594429 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:23.594133    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:23.594429 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:23.594200    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:24.589903 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:24.589860    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:24.590095 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:24.589999    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:25.591881 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:25.591817    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:25.592343 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:25.591939    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:25.592343 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:25.592006    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:25.592343 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:25.592128    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:26.590045 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:26.589989    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:26.590245 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:26.590143    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:27.590361 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:27.590315    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:27.590809 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:27.590318    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:27.590809 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:27.590439    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:27.590809 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:27.590534    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:28.589874 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:28.589832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:28.590115 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:28.589984    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:29.243458 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:29.243420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:29.243955 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.243591    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:29.243955 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.243618    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:29.243955 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.243632    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rq4kd for pod openshift-network-diagnostics/network-check-target-44vpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:29.243955 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.243701    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd podName:374c967a-24a3-40b4-872f-bfa954850357 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:45.243683641 +0000 UTC m=+34.172587384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rq4kd" (UniqueName: "kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd") pod "network-check-target-44vpt" (UID: "374c967a-24a3-40b4-872f-bfa954850357") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:29.343778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:29.343727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:29.343778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:29.343779    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:29.344045 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.343917    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:29.344045 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.343997    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret podName:de8493b4-7db2-438d-bcaa-b97e00b8c2e3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:45.343974932 +0000 UTC m=+34.272878671 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret") pod "global-pull-secret-syncer-jr9bb" (UID: "de8493b4-7db2-438d-bcaa-b97e00b8c2e3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:29.344179 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.343921    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:29.344179 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.344109    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs podName:29de17e1-0cb6-41fb-bd61-0a1fc39b19b3 nodeName:}" failed. No retries permitted until 2026-09-08 13:16:45.344091149 +0000 UTC m=+34.272994883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs") pod "network-metrics-daemon-82vzh" (UID: "29de17e1-0cb6-41fb-bd61-0a1fc39b19b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:29.589690 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:29.589653    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:29.589877 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:29.589653    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:29.589877 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.589785    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:29.590039 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:29.589893    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:30.589533 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:30.589498    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:30.590002 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:30.589636    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:31.589885 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:31.589847    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:31.590348 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:31.589926    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:31.590348 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:31.590029    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:31.590348 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:31.590152    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:32.589980 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:32.589947    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:32.590422 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:32.590113    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:33.590415 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.590245    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:33.591059 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.590324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:33.591059 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:33.590507    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:33.591059 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:33.590617    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:33.694533 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.694498    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l9frn" event={"ID":"02703e20-c856-4652-9e02-6086e7b11179","Type":"ContainerStarted","Data":"659919f73547a720c8d3a64bc20459ab718aeaf9eb38dacbbb5b6b1cef951ee8"}
Sep 08 13:16:33.696152 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.696111    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ch66t" event={"ID":"a4b6680b-cc21-46ee-a842-48b689876d97","Type":"ContainerStarted","Data":"6e30a51bfc5b8a98cc1efdce5cadb96439a6583d91e464100b9f1bde844fa52d"}
Sep 08 13:16:33.697615 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.697589    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-78brt" event={"ID":"92745d4d-0c99-4d1b-a91e-b7a546e0e583","Type":"ContainerStarted","Data":"1cf06a8d10ebe799a9001417d33772b99db6a447b912df6df842199e35e45282"}
Sep 08 13:16:33.699072 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.699047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj" event={"ID":"a1afd7b4-babf-47e5-b128-300b8dde301f","Type":"ContainerStarted","Data":"8195a4ef2208a9d118667610e18e9453d0598cf4b751fbe7472bae43f53eb91e"}
Sep 08 13:16:33.700453 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.700429    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vrfh7" event={"ID":"817efad1-a9dc-4e98-ae2d-396f97d0eda9","Type":"ContainerStarted","Data":"2dbb35e32a11767d50d10669fba036124d315d055ce6e20265fc1c3dbe11d141"}
Sep 08 13:16:33.701852 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.701820    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-szhm7" event={"ID":"d533862a-b3ec-4f39-9018-e357ba3f02c5","Type":"ContainerStarted","Data":"1c8d6391591fc793df4a14d032a0c178957f9f400524a0d70c49fee9c89a5f1d"}
Sep 08 13:16:33.703348 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.703326    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5dbb570-2303-4ba8-848c-b8ac08d60865" containerID="79a5d8d8b1f6de197e8a6cf1c5960253032b5e36d8ddabf2556431bc6e3f4746" exitCode=0
Sep 08 13:16:33.703424 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.703402    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerDied","Data":"79a5d8d8b1f6de197e8a6cf1c5960253032b5e36d8ddabf2556431bc6e3f4746"}
Sep 08 13:16:33.710190 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.710168    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" event={"ID":"5d710a10-f153-4cad-9c8f-2482d1a4ebef","Type":"ContainerStarted","Data":"2940c01014212b8c706909fbc39a8c6e401fde5d18c3361ba6adbb60d0f5d966"}
Sep 08 13:16:33.710281 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.710245    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" event={"ID":"5d710a10-f153-4cad-9c8f-2482d1a4ebef","Type":"ContainerStarted","Data":"b5833a89f17ea188515648dcf0d27f14eb39834d2aaa239bcda9c974635ca189"}
Sep 08 13:16:33.710333 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.710292    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" event={"ID":"5d710a10-f153-4cad-9c8f-2482d1a4ebef","Type":"ContainerStarted","Data":"aba0372dec6b29f68aefa9604da74bfa327e9a874ff0a498dfd6887471daff24"}
Sep 08 13:16:33.710372 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.710338    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" event={"ID":"5d710a10-f153-4cad-9c8f-2482d1a4ebef","Type":"ContainerStarted","Data":"62a35890df1ff271197269dffb96f0a897d0d429de9a4c561db3cd594b6b2fa3"}
Sep 08 13:16:33.710403 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.710382    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" event={"ID":"5d710a10-f153-4cad-9c8f-2482d1a4ebef","Type":"ContainerStarted","Data":"c340f17c3ab8452c65c919c5cc2318619cc27db1a262c5ac54c30bb117153531"}
Sep 08 13:16:33.710442 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.710431    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" event={"ID":"5d710a10-f153-4cad-9c8f-2482d1a4ebef","Type":"ContainerStarted","Data":"28e3ece13aeb04a0f1f6cfe79061f3543e2e7807f2f6f0339dbf6dfa0c4cca98"}
Sep 08 13:16:33.713050 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.712994    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-66.ec2.internal" podStartSLOduration=20.712981962 podStartE2EDuration="20.712981962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:16.674585828 +0000 UTC m=+5.603489586" watchObservedRunningTime="2026-09-08 13:16:33.712981962 +0000 UTC m=+22.641885702"
Sep 08 13:16:33.728473 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.728438    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l9frn" podStartSLOduration=3.6435451519999997 podStartE2EDuration="21.72842492s" totalImagesPullingTime="18.084879768s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:33.713544406 +0000 UTC m=+22.642448147" watchObservedRunningTime="2026-09-08 13:16:33.72842492 +0000 UTC m=+22.657328662"
Sep 08 13:16:33.728707 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.728688    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-78brt" podStartSLOduration=3.681671415 podStartE2EDuration="21.728680257s" totalImagesPullingTime="18.047008842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:33.728368154 +0000 UTC m=+22.657271896" watchObservedRunningTime="2026-09-08 13:16:33.728680257 +0000 UTC m=+22.657583998"
Sep 08 13:16:33.745118 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.745077    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-szhm7" podStartSLOduration=3.638660205 podStartE2EDuration="21.74506427s" totalImagesPullingTime="18.106404065s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:33.744417031 +0000 UTC m=+22.673320772" watchObservedRunningTime="2026-09-08 13:16:33.74506427 +0000 UTC m=+22.673968012"
Sep 08 13:16:33.778963 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:33.778880    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ch66t" podStartSLOduration=3.717796833 podStartE2EDuration="21.778865722s" totalImagesPullingTime="18.061068889s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:33.77845565 +0000 UTC m=+22.707359392" watchObservedRunningTime="2026-09-08 13:16:33.778865722 +0000 UTC m=+22.707769469"
Sep 08 13:16:34.440710 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:34.440675    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:34.441461 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:34.441436    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:34.457516 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:34.457463    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vrfh7" podStartSLOduration=5.356877214 podStartE2EDuration="22.457446595s" totalImagesPullingTime="17.100569381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:33.79342907 +0000 UTC m=+22.722332823" watchObservedRunningTime="2026-09-08 13:16:34.457446595 +0000 UTC m=+23.386350338"
Sep 08 13:16:34.590230 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:34.590157    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:34.590392 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:34.590298    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:34.715253 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:34.715152    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-98nf9" event={"ID":"b546af0c-d8b5-4ce6-a934-52eeab3ef44d","Type":"ContainerStarted","Data":"631374c3cd8db3ffd891f05273e9bac55cedc1750fbe42cd080edd7e8fc5acc6"}
Sep 08 13:16:34.716125 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:34.715862    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:34.716125 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:34.715923    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-78brt"
Sep 08 13:16:34.746193 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:34.746128    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-98nf9" podStartSLOduration=4.689699289 podStartE2EDuration="22.746106865s" totalImagesPullingTime="18.056407576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:34.730768913 +0000 UTC m=+23.659672655" watchObservedRunningTime="2026-09-08 13:16:34.746106865 +0000 UTC m=+23.675010609"
Sep 08 13:16:35.590117 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:35.589570    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:35.590117 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:35.589695    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:35.590117 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:35.590045    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:35.590421 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:35.590158    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:36.216379 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:36.216201    2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 13:16:36.559718 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:36.559530    2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T13:16:36.216375969Z","UUID":"187ea1dd-d0eb-4ea9-aa80-195db615c06c","Handler":null,"Name":"","Endpoint":""}
Sep 08 13:16:36.561093 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:36.561075    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 13:16:36.561093 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:36.561104    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 13:16:36.589710 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:36.589677    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:36.589845 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:36.589779    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:36.723593 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:36.723553    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" event={"ID":"5d710a10-f153-4cad-9c8f-2482d1a4ebef","Type":"ContainerStarted","Data":"f643e56ae5ee59e7f80b3bbe91efcf1f27a868e95aa3b3b363a46877322ccbae"}
Sep 08 13:16:36.725481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:36.725455    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj" event={"ID":"a1afd7b4-babf-47e5-b128-300b8dde301f","Type":"ContainerStarted","Data":"973e75ed39e2173a08d84cfac91e4ad87fb17fba2fbed62f5d964e35593a7324"}
Sep 08 13:16:37.590112 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:37.590074    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:37.590568 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:37.590075    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:37.590568 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:37.590218    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:37.590568 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:37.590291    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:38.590304 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.590118    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:38.590779 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:38.590416    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:38.732405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.732327    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5dbb570-2303-4ba8-848c-b8ac08d60865" containerID="12c37a0a9790d037efe7b1621f1eedfc13869787f55d10843bbad2dde9b7eb26" exitCode=0
Sep 08 13:16:38.732538 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.732411    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerDied","Data":"12c37a0a9790d037efe7b1621f1eedfc13869787f55d10843bbad2dde9b7eb26"}
Sep 08 13:16:38.736443 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.736416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" event={"ID":"5d710a10-f153-4cad-9c8f-2482d1a4ebef","Type":"ContainerStarted","Data":"06fa18f1a5a34068842f963789f6f81955745754300b7aaa909fb831a4dc19a4"}
Sep 08 13:16:38.736729 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.736697    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:38.736822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.736805    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:38.736876 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.736855    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:38.742812 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.742778    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj" event={"ID":"a1afd7b4-babf-47e5-b128-300b8dde301f","Type":"ContainerStarted","Data":"766ecaeaf3749adb500bb622b20ae4f1819d2b9e07b236fc4901ca6ee007c623"}
Sep 08 13:16:38.753643 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.753619    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:38.754332 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.754310    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:16:38.803822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.803764    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc" podStartSLOduration=8.677478309 podStartE2EDuration="26.803744778s" totalImagesPullingTime="18.126266469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:38.803190026 +0000 UTC m=+27.732093787" watchObservedRunningTime="2026-09-08 13:16:38.803744778 +0000 UTC m=+27.732648519"
Sep 08 13:16:38.803977 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:38.803957    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zftxj" podStartSLOduration=3.762166304 podStartE2EDuration="26.803951987s" totalImagesPullingTime="23.041785683s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:38.775792713 +0000 UTC m=+27.704696475" watchObservedRunningTime="2026-09-08 13:16:38.803951987 +0000 UTC m=+27.732855729"
Sep 08 13:16:39.592737 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:39.592708    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:39.593371 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:39.592803    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:39.593371 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:39.592853    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:39.593371 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:39.592952    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:40.071183 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:40.071150    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-44vpt"]
Sep 08 13:16:40.071328 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:40.071256    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:40.071370 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:40.071346    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:40.074277 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:40.074242    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jr9bb"]
Sep 08 13:16:40.074438 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:40.074353    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:40.074514 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:40.074446    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:40.075025 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:40.074980    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-82vzh"]
Sep 08 13:16:40.075142 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:40.075111    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:40.075250 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:40.075227    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:40.749651 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:40.749610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerStarted","Data":"1e554b5685da421c8a62a8384d603ba5ce7d8d06e13344a4126e0d94d86360a5"}
Sep 08 13:16:41.589898 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:41.589866    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:41.590072 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:41.589951    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:41.590072 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:41.590023    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:41.590072 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:41.590033    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:41.590186 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:41.590118    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:41.590220 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:41.590186    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:41.753784 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:41.753750    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5dbb570-2303-4ba8-848c-b8ac08d60865" containerID="1e554b5685da421c8a62a8384d603ba5ce7d8d06e13344a4126e0d94d86360a5" exitCode=0
Sep 08 13:16:41.754188 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:41.753811    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerDied","Data":"1e554b5685da421c8a62a8384d603ba5ce7d8d06e13344a4126e0d94d86360a5"}
Sep 08 13:16:43.590024 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:43.589755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:43.590425 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:43.589762    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:43.590425 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:43.590153    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:43.590425 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:43.589762    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:43.590425 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:43.590220    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:43.590425 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:43.590265    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:45.253890 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:45.253857    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:45.254403 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.254034    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 13:16:45.254403 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.254055    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 13:16:45.254403 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.254064    2583 projected.go:196] Error preparing data for projected volume kube-api-access-rq4kd for pod openshift-network-diagnostics/network-check-target-44vpt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:45.254403 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.254112    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd podName:374c967a-24a3-40b4-872f-bfa954850357 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:17.254098913 +0000 UTC m=+66.183002632 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rq4kd" (UniqueName: "kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd") pod "network-check-target-44vpt" (UID: "374c967a-24a3-40b4-872f-bfa954850357") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 13:16:45.354604 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:45.354570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:45.354604 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:45.354606    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:45.354853 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.354699    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:45.354853 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.354727    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:45.354853 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.354752    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs podName:29de17e1-0cb6-41fb-bd61-0a1fc39b19b3 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:17.354739025 +0000 UTC m=+66.283642745 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs") pod "network-metrics-daemon-82vzh" (UID: "29de17e1-0cb6-41fb-bd61-0a1fc39b19b3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 13:16:45.354853 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.354789    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret podName:de8493b4-7db2-438d-bcaa-b97e00b8c2e3 nodeName:}" failed. No retries permitted until 2026-09-08 13:17:17.354771092 +0000 UTC m=+66.283674825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret") pod "global-pull-secret-syncer-jr9bb" (UID: "de8493b4-7db2-438d-bcaa-b97e00b8c2e3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 13:16:45.589651 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:45.589618    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:45.589651 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:45.589647    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:45.589895 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:45.589738    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:45.589895 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.589755    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-44vpt" podUID="374c967a-24a3-40b4-872f-bfa954850357"
Sep 08 13:16:45.589895 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.589849    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:16:45.590061 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:45.589933    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jr9bb" podUID="de8493b4-7db2-438d-bcaa-b97e00b8c2e3"
Sep 08 13:16:46.766422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.766386    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerStarted","Data":"045a73ab12ffa94e29b26891a1a90c6b240f6b7006c0935f1a44d6f939125237"}
Sep 08 13:16:46.845328 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.845302    2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-66.ec2.internal" event="NodeReady"
Sep 08 13:16:46.845527 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.845455    2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 13:16:46.884570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.884538    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7789bf9865-vzcqq"]
Sep 08 13:16:46.897827 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.897796    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:46.901961 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.901939    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 13:16:46.902106 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.902001    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 13:16:46.902106 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.902079    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-55db4\""
Sep 08 13:16:46.902106 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.902082    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 13:16:46.908002 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.904426    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k6zh9"]
Sep 08 13:16:46.910555 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.910536    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 13:16:46.929959 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.929925    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9qg7h"]
Sep 08 13:16:46.930123 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.930063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:46.932356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.932329    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 13:16:46.932839 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.932807    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 13:16:46.932839 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.932807    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kwlvq\""
Sep 08 13:16:46.953871 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.953843    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7789bf9865-vzcqq"]
Sep 08 13:16:46.953975 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.953895    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9qg7h"]
Sep 08 13:16:46.953975 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.953911    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k6zh9"]
Sep 08 13:16:46.953975 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.953846    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9qg7h"
Sep 08 13:16:46.956193 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.956175    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 13:16:46.956308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.956291    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 13:16:46.956363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.956334    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t98tn\""
Sep 08 13:16:46.956559 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.956547    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 13:16:46.965674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.965652    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-installation-pull-secrets\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:46.965769 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.965682    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-bound-sa-token\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:46.965769 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.965703    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-certificates\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:46.965859 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.965765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:46.965859 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.965798    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh82c\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-kube-api-access-xh82c\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:46.965960 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.965911    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-image-registry-private-configuration\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:46.965960 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.965945    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-ca-trust-extracted\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:46.966075 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:46.965981    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-trusted-ca\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.066483 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066451    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.066637 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066487    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh82c\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-kube-api-access-xh82c\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.066637 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066510    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6529bc-e29e-406f-ae0e-55c85d81465b-config-volume\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.066637 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066583    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6529bc-e29e-406f-ae0e-55c85d81465b-tmp-dir\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.066637 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066621    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4mql\" (UniqueName: \"kubernetes.io/projected/8e6529bc-e29e-406f-ae0e-55c85d81465b-kube-api-access-c4mql\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.066762 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066664    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a352c5b0-060c-4938-9775-9138f5461a28-cert\") pod \"ingress-canary-9qg7h\" (UID: \"a352c5b0-060c-4938-9775-9138f5461a28\") " pod="openshift-ingress-canary/ingress-canary-9qg7h"
Sep 08 13:16:47.066762 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066706    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-image-registry-private-configuration\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.066762 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066725    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-ca-trust-extracted\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.066762 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.066723    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:47.066762 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.066760    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7789bf9865-vzcqq: secret "image-registry-tls" not found
Sep 08 13:16:47.066930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-trusted-ca\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.066930 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.066829    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls podName:12f11884-1f2b-49bf-99b4-4c7f5f299fbe nodeName:}" failed. No retries permitted until 2026-09-08 13:16:47.566807731 +0000 UTC m=+36.495711451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls") pod "image-registry-7789bf9865-vzcqq" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe") : secret "image-registry-tls" not found
Sep 08 13:16:47.066930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066853    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9htq\" (UniqueName: \"kubernetes.io/projected/a352c5b0-060c-4938-9775-9138f5461a28-kube-api-access-p9htq\") pod \"ingress-canary-9qg7h\" (UID: \"a352c5b0-060c-4938-9775-9138f5461a28\") " pod="openshift-ingress-canary/ingress-canary-9qg7h"
Sep 08 13:16:47.066930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066890    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-installation-pull-secrets\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.066930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066914    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-bound-sa-token\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.067343 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066946    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-certificates\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.067343 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.066977    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.067343 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.067235    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-ca-trust-extracted\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.067659 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.067609    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-certificates\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.067865 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.067847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-trusted-ca\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.070839 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.070816    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-image-registry-private-configuration\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.070920 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.070820    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-installation-pull-secrets\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.075428 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.075404    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-bound-sa-token\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.075670 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.075650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh82c\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-kube-api-access-xh82c\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.167480 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.167441    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6529bc-e29e-406f-ae0e-55c85d81465b-config-volume\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.167480 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.167483    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6529bc-e29e-406f-ae0e-55c85d81465b-tmp-dir\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.167722 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.167505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4mql\" (UniqueName: \"kubernetes.io/projected/8e6529bc-e29e-406f-ae0e-55c85d81465b-kube-api-access-c4mql\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.167722 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.167540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a352c5b0-060c-4938-9775-9138f5461a28-cert\") pod \"ingress-canary-9qg7h\" (UID: \"a352c5b0-060c-4938-9775-9138f5461a28\") " pod="openshift-ingress-canary/ingress-canary-9qg7h"
Sep 08 13:16:47.167722 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.167585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9htq\" (UniqueName: \"kubernetes.io/projected/a352c5b0-060c-4938-9775-9138f5461a28-kube-api-access-p9htq\") pod \"ingress-canary-9qg7h\" (UID: \"a352c5b0-060c-4938-9775-9138f5461a28\") " pod="openshift-ingress-canary/ingress-canary-9qg7h"
Sep 08 13:16:47.167722 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.167628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.167911 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.167728    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:47.167911 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.167795    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls podName:8e6529bc-e29e-406f-ae0e-55c85d81465b nodeName:}" failed. No retries permitted until 2026-09-08 13:16:47.667776695 +0000 UTC m=+36.596680429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls") pod "dns-default-k6zh9" (UID: "8e6529bc-e29e-406f-ae0e-55c85d81465b") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:47.167911 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.167879    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e6529bc-e29e-406f-ae0e-55c85d81465b-tmp-dir\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.168100 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.168081    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e6529bc-e29e-406f-ae0e-55c85d81465b-config-volume\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.170158 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.170140    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a352c5b0-060c-4938-9775-9138f5461a28-cert\") pod \"ingress-canary-9qg7h\" (UID: \"a352c5b0-060c-4938-9775-9138f5461a28\") " pod="openshift-ingress-canary/ingress-canary-9qg7h"
Sep 08 13:16:47.178462 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.178435    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9htq\" (UniqueName: \"kubernetes.io/projected/a352c5b0-060c-4938-9775-9138f5461a28-kube-api-access-p9htq\") pod \"ingress-canary-9qg7h\" (UID: \"a352c5b0-060c-4938-9775-9138f5461a28\") " pod="openshift-ingress-canary/ingress-canary-9qg7h"
Sep 08 13:16:47.179037 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.179006    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4mql\" (UniqueName: \"kubernetes.io/projected/8e6529bc-e29e-406f-ae0e-55c85d81465b-kube-api-access-c4mql\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.262488 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.262448    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9qg7h"
Sep 08 13:16:47.310611 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.310577    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:47.571136 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.570950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:47.571280 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.571113    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:47.571280 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.571201    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7789bf9865-vzcqq: secret "image-registry-tls" not found
Sep 08 13:16:47.571280 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.571253    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls podName:12f11884-1f2b-49bf-99b4-4c7f5f299fbe nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.571239316 +0000 UTC m=+37.500143049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls") pod "image-registry-7789bf9865-vzcqq" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe") : secret "image-registry-tls" not found
Sep 08 13:16:47.582742 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.582709    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9qg7h"]
Sep 08 13:16:47.586611 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:16:47.586587    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda352c5b0_060c_4938_9775_9138f5461a28.slice/crio-2dbc9b5e0687bbe8bb58db9de789028eea30d519c43cca061cbfdae65b5d9c53 WatchSource:0}: Error finding container 2dbc9b5e0687bbe8bb58db9de789028eea30d519c43cca061cbfdae65b5d9c53: Status 404 returned error can't find the container with id 2dbc9b5e0687bbe8bb58db9de789028eea30d519c43cca061cbfdae65b5d9c53
Sep 08 13:16:47.589954 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.589932    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:16:47.590067 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.589932    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:16:47.590132 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.589934    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:16:47.592347 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.592282    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:16:47.592347 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.592336    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g794g\""
Sep 08 13:16:47.592552 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.592368    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:16:47.592552 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.592394    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:16:47.592717 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.592699    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mxg9n\""
Sep 08 13:16:47.592828 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.592727    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:16:47.672294 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.672203    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:47.672466 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.672387    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:47.672505 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:47.672469    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls podName:8e6529bc-e29e-406f-ae0e-55c85d81465b nodeName:}" failed. No retries permitted until 2026-09-08 13:16:48.672450374 +0000 UTC m=+37.601354096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls") pod "dns-default-k6zh9" (UID: "8e6529bc-e29e-406f-ae0e-55c85d81465b") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:47.771132 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.771094    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5dbb570-2303-4ba8-848c-b8ac08d60865" containerID="045a73ab12ffa94e29b26891a1a90c6b240f6b7006c0935f1a44d6f939125237" exitCode=0
Sep 08 13:16:47.771620 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.771186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerDied","Data":"045a73ab12ffa94e29b26891a1a90c6b240f6b7006c0935f1a44d6f939125237"}
Sep 08 13:16:47.772397 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:47.772367    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9qg7h" event={"ID":"a352c5b0-060c-4938-9775-9138f5461a28","Type":"ContainerStarted","Data":"2dbc9b5e0687bbe8bb58db9de789028eea30d519c43cca061cbfdae65b5d9c53"}
Sep 08 13:16:48.579944 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:48.579904    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:48.580176 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:48.580091    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:48.580176 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:48.580112    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7789bf9865-vzcqq: secret "image-registry-tls" not found
Sep 08 13:16:48.580284 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:48.580192    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls podName:12f11884-1f2b-49bf-99b4-4c7f5f299fbe nodeName:}" failed. No retries permitted until 2026-09-08 13:16:50.580172356 +0000 UTC m=+39.509076076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls") pod "image-registry-7789bf9865-vzcqq" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe") : secret "image-registry-tls" not found
Sep 08 13:16:48.680617 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:48.680571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:48.680813 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:48.680735    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:48.680813 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:48.680811    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls podName:8e6529bc-e29e-406f-ae0e-55c85d81465b nodeName:}" failed. No retries permitted until 2026-09-08 13:16:50.680787103 +0000 UTC m=+39.609690881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls") pod "dns-default-k6zh9" (UID: "8e6529bc-e29e-406f-ae0e-55c85d81465b") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:50.598192 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:50.598147    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:50.598792 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:50.598305    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:50.598792 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:50.598325    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7789bf9865-vzcqq: secret "image-registry-tls" not found
Sep 08 13:16:50.598792 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:50.598382    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls podName:12f11884-1f2b-49bf-99b4-4c7f5f299fbe nodeName:}" failed. No retries permitted until 2026-09-08 13:16:54.598366944 +0000 UTC m=+43.527270683 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls") pod "image-registry-7789bf9865-vzcqq" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe") : secret "image-registry-tls" not found
Sep 08 13:16:50.698687 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:50.698652    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:50.698886 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:50.698801    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:50.698886 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:50.698868    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls podName:8e6529bc-e29e-406f-ae0e-55c85d81465b nodeName:}" failed. No retries permitted until 2026-09-08 13:16:54.698850013 +0000 UTC m=+43.627753748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls") pod "dns-default-k6zh9" (UID: "8e6529bc-e29e-406f-ae0e-55c85d81465b") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:50.790233 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:50.790191    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:50.849166 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:50.849072    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:51.460593 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:51.460561    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:51.533363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:51.533325    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:52.789312 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:52.789275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerStarted","Data":"75ffa3dc9cbc0b6ec873cd18d5bbd12cdf8f9e21a97dd0f73cc5af2d9802814c"}
Sep 08 13:16:53.795778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:53.795746    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5dbb570-2303-4ba8-848c-b8ac08d60865" containerID="75ffa3dc9cbc0b6ec873cd18d5bbd12cdf8f9e21a97dd0f73cc5af2d9802814c" exitCode=0
Sep 08 13:16:53.795778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:53.795789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerDied","Data":"75ffa3dc9cbc0b6ec873cd18d5bbd12cdf8f9e21a97dd0f73cc5af2d9802814c"}
Sep 08 13:16:54.630069 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:54.630020    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:16:54.630221 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:54.630135    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:16:54.630221 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:54.630158    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7789bf9865-vzcqq: secret "image-registry-tls" not found
Sep 08 13:16:54.630221 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:54.630216    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls podName:12f11884-1f2b-49bf-99b4-4c7f5f299fbe nodeName:}" failed. No retries permitted until 2026-09-08 13:17:02.630200262 +0000 UTC m=+51.559103997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls") pod "image-registry-7789bf9865-vzcqq" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe") : secret "image-registry-tls" not found
Sep 08 13:16:54.730780 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:54.730701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:16:54.730905 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:54.730857    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:16:54.730939 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:16:54.730918    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls podName:8e6529bc-e29e-406f-ae0e-55c85d81465b nodeName:}" failed. No retries permitted until 2026-09-08 13:17:02.730902524 +0000 UTC m=+51.659806243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls") pod "dns-default-k6zh9" (UID: "8e6529bc-e29e-406f-ae0e-55c85d81465b") : secret "dns-default-metrics-tls" not found
Sep 08 13:16:54.801335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:54.801301    2583 generic.go:356] "Generic (PLEG): container finished" podID="a5dbb570-2303-4ba8-848c-b8ac08d60865" containerID="96f08a196cbdc1ec56ae6106a270618c836f45864a2bbf66726d3547ddfbf71b" exitCode=0
Sep 08 13:16:54.801749 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:54.801376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerDied","Data":"96f08a196cbdc1ec56ae6106a270618c836f45864a2bbf66726d3547ddfbf71b"}
Sep 08 13:16:54.802851 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:54.802824    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9qg7h" event={"ID":"a352c5b0-060c-4938-9775-9138f5461a28","Type":"ContainerStarted","Data":"99a9280744dc1852e7d4205d8c16eafc3a7db253e1843837aba66140300dfce1"}
Sep 08 13:16:54.839874 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:54.839708    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9qg7h" podStartSLOduration=1.95876497 podStartE2EDuration="8.8396881s" totalImagesPullingTime="6.88092313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:54.839580865 +0000 UTC m=+43.768484608" watchObservedRunningTime="2026-09-08 13:16:54.8396881 +0000 UTC m=+43.768591820"
Sep 08 13:16:55.808222 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:55.808186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" event={"ID":"a5dbb570-2303-4ba8-848c-b8ac08d60865","Type":"ContainerStarted","Data":"9f82b592dc31e24cf472dbe77a0367f70a7e79408a04ff0b3c9a92d4b1534602"}
Sep 08 13:16:55.809207 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:55.809187    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:55.831917 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:55.831872    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mw5r7" podStartSLOduration=10.133395949 podStartE2EDuration="43.831858177s" totalImagesPullingTime="33.698462228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:16:55.831333558 +0000 UTC m=+44.760237299" watchObservedRunningTime="2026-09-08 13:16:55.831858177 +0000 UTC m=+44.760761942"
Sep 08 13:16:55.909705 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:55.909672    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:56.009941 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:56.009906    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:56.070308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:56.070277    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:58.833819 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:58.833783    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:58.915048 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:58.915000    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:59.137192 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:59.137111    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:16:59.745967 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:16:59.745938    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:02.690902 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:02.690845    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:17:02.691368 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:02.690906    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:17:02.691368 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:02.690949    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7789bf9865-vzcqq: secret "image-registry-tls" not found
Sep 08 13:17:02.691368 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:02.691050    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls podName:12f11884-1f2b-49bf-99b4-4c7f5f299fbe nodeName:}" failed. No retries permitted until 2026-09-08 13:17:18.691004301 +0000 UTC m=+67.619908089 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls") pod "image-registry-7789bf9865-vzcqq" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe") : secret "image-registry-tls" not found
Sep 08 13:17:02.792870 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:02.792829    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:17:02.793053 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:02.792981    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:02.793098 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:02.793081    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls podName:8e6529bc-e29e-406f-ae0e-55c85d81465b nodeName:}" failed. No retries permitted until 2026-09-08 13:17:18.793063751 +0000 UTC m=+67.721967472 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls") pod "dns-default-k6zh9" (UID: "8e6529bc-e29e-406f-ae0e-55c85d81465b") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:06.896332 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:06.896294    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:06.961491 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:06.961462    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:07.841055 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:07.841005    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:07.940300 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:07.940270    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:09.775550 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:09.775514    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:10.760816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:10.760788    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b2fbc"
Sep 08 13:17:10.912023 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:10.911990    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:10.972694 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:10.972660    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:17.295672 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.295608    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:17:17.298309 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.298293    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 13:17:17.308580 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.308558    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 13:17:17.322559 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.320066    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4kd\" (UniqueName: \"kubernetes.io/projected/374c967a-24a3-40b4-872f-bfa954850357-kube-api-access-rq4kd\") pod \"network-check-target-44vpt\" (UID: \"374c967a-24a3-40b4-872f-bfa954850357\") " pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:17:17.396640 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.396584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:17:17.396640 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.396643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:17:17.398985 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.398965    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 13:17:17.399173 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.399162    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 13:17:17.407631 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:17.407613    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:17:17.407735 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:17.407675    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs podName:29de17e1-0cb6-41fb-bd61-0a1fc39b19b3 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:21.407659078 +0000 UTC m=+130.336562798 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs") pod "network-metrics-daemon-82vzh" (UID: "29de17e1-0cb6-41fb-bd61-0a1fc39b19b3") : secret "metrics-daemon-secret" not found
Sep 08 13:17:17.410113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.410084    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/de8493b4-7db2-438d-bcaa-b97e00b8c2e3-original-pull-secret\") pod \"global-pull-secret-syncer-jr9bb\" (UID: \"de8493b4-7db2-438d-bcaa-b97e00b8c2e3\") " pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:17:17.600427 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.600398    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jr9bb"
Sep 08 13:17:17.607204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.607183    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-g794g\""
Sep 08 13:17:17.615973 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.615950    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:17:17.869797 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.869714    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jr9bb"]
Sep 08 13:17:17.873210 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:17:17.873187    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde8493b4_7db2_438d_bcaa_b97e00b8c2e3.slice/crio-dafd1e23b248b28210ccb36034e421c245c40d235fff3963c9282182205cc0ae WatchSource:0}: Error finding container dafd1e23b248b28210ccb36034e421c245c40d235fff3963c9282182205cc0ae: Status 404 returned error can't find the container with id dafd1e23b248b28210ccb36034e421c245c40d235fff3963c9282182205cc0ae
Sep 08 13:17:17.892355 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:17.892325    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-44vpt"]
Sep 08 13:17:17.895907 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:17:17.895878    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374c967a_24a3_40b4_872f_bfa954850357.slice/crio-6f54933df649243e84a45e67aaf97447c40e20af140aa51d84aac0ba89f62e7a WatchSource:0}: Error finding container 6f54933df649243e84a45e67aaf97447c40e20af140aa51d84aac0ba89f62e7a: Status 404 returned error can't find the container with id 6f54933df649243e84a45e67aaf97447c40e20af140aa51d84aac0ba89f62e7a
Sep 08 13:17:18.707287 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:18.707243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:17:18.707775 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:18.707402    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:17:18.707775 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:18.707425    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7789bf9865-vzcqq: secret "image-registry-tls" not found
Sep 08 13:17:18.707775 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:18.707482    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls podName:12f11884-1f2b-49bf-99b4-4c7f5f299fbe nodeName:}" failed. No retries permitted until 2026-09-08 13:17:50.707467221 +0000 UTC m=+99.636370957 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls") pod "image-registry-7789bf9865-vzcqq" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe") : secret "image-registry-tls" not found
Sep 08 13:17:18.807912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:18.807868    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:17:18.808130 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:18.808059    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:18.808201 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:18.808144    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls podName:8e6529bc-e29e-406f-ae0e-55c85d81465b nodeName:}" failed. No retries permitted until 2026-09-08 13:17:50.808121077 +0000 UTC m=+99.737024813 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls") pod "dns-default-k6zh9" (UID: "8e6529bc-e29e-406f-ae0e-55c85d81465b") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:18.863384 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:18.863346    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-44vpt" event={"ID":"374c967a-24a3-40b4-872f-bfa954850357","Type":"ContainerStarted","Data":"6f54933df649243e84a45e67aaf97447c40e20af140aa51d84aac0ba89f62e7a"}
Sep 08 13:17:18.864550 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:18.864511    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jr9bb" event={"ID":"de8493b4-7db2-438d-bcaa-b97e00b8c2e3","Type":"ContainerStarted","Data":"dafd1e23b248b28210ccb36034e421c245c40d235fff3963c9282182205cc0ae"}
Sep 08 13:17:19.797364 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:19.797330    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:19.896393 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:19.896322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:21.876329 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:21.876290    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-44vpt" event={"ID":"374c967a-24a3-40b4-872f-bfa954850357","Type":"ContainerStarted","Data":"317c9250511d489cdc28861653417090fb8407c185595eb9d58978907e774c80"}
Sep 08 13:17:21.876859 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:21.876376    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:17:21.891791 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:21.891728    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-44vpt" podStartSLOduration=66.963804012 podStartE2EDuration="1m9.891704482s" totalImagesPullingTime="2.92790047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:21.891544853 +0000 UTC m=+70.820448611" watchObservedRunningTime="2026-09-08 13:17:21.891704482 +0000 UTC m=+70.820608252"
Sep 08 13:17:22.918254 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:22.918223    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:23.016433 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:23.016404    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:23.882588 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:23.882556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jr9bb" event={"ID":"de8493b4-7db2-438d-bcaa-b97e00b8c2e3","Type":"ContainerStarted","Data":"a902bd365f6c5e697c23e903becb13a896564473de27dca9d698e14fed541566"}
Sep 08 13:17:23.898028 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:23.897962    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jr9bb" podStartSLOduration=66.913948268 podStartE2EDuration="1m11.897945151s" totalImagesPullingTime="4.983996883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:17:23.897254182 +0000 UTC m=+72.826157924" watchObservedRunningTime="2026-09-08 13:17:23.897945151 +0000 UTC m=+72.826848892"
Sep 08 13:17:24.498196 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:24.498168    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:24.576356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:24.576330    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:25.545779 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:25.545744    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:25.646805 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:25.646772    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:45.885814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:45.885658    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:45.984260 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:45.984226    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:17:50.747911 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:50.747869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:17:50.748314 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:50.747998    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:17:50.748314 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:50.748027    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7789bf9865-vzcqq: secret "image-registry-tls" not found
Sep 08 13:17:50.748314 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:50.748081    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls podName:12f11884-1f2b-49bf-99b4-4c7f5f299fbe nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.748066417 +0000 UTC m=+163.676970153 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls") pod "image-registry-7789bf9865-vzcqq" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe") : secret "image-registry-tls" not found
Sep 08 13:17:50.849214 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:50.849165    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:17:50.849386 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:50.849319    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:17:50.849428 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:17:50.849397    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls podName:8e6529bc-e29e-406f-ae0e-55c85d81465b nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.849381249 +0000 UTC m=+163.778284987 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls") pod "dns-default-k6zh9" (UID: "8e6529bc-e29e-406f-ae0e-55c85d81465b") : secret "dns-default-metrics-tls" not found
Sep 08 13:17:52.880979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:17:52.880940    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-44vpt"
Sep 08 13:18:21.471560 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:21.471506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:18:21.472080 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:21.471631    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 13:18:21.472080 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:21.471696    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs podName:29de17e1-0cb6-41fb-bd61-0a1fc39b19b3 nodeName:}" failed. No retries permitted until 2026-09-08 13:20:23.47168134 +0000 UTC m=+252.400585060 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs") pod "network-metrics-daemon-82vzh" (UID: "29de17e1-0cb6-41fb-bd61-0a1fc39b19b3") : secret "metrics-daemon-secret" not found
Sep 08 13:18:40.912836 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:40.912782    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:49.908808 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:49.908717    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7789bf9865-vzcqq" podUID="12f11884-1f2b-49bf-99b4-4c7f5f299fbe"
Sep 08 13:18:49.939111 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:49.939067    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-k6zh9" podUID="8e6529bc-e29e-406f-ae0e-55c85d81465b"
Sep 08 13:18:50.091799 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:50.091772    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:18:50.091952 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:50.091776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:18:50.609823 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:50.609776    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-82vzh" podUID="29de17e1-0cb6-41fb-bd61-0a1fc39b19b3"
Sep 08 13:18:54.022638 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.022595    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8rdfw"]
Sep 08 13:18:54.024495 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.024478    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-6lrnr"]
Sep 08 13:18:54.024617 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.024598    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.026228 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.026202    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-567cf589fd-rkshv"]
Sep 08 13:18:54.026341 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.026300    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.027922 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.027903    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.028499 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.028476    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 08 13:18:54.028499 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.028495    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dq5h6\""
Sep 08 13:18:54.028674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.028616    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 08 13:18:54.029184 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.029167    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 13:18:54.029287 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.029200    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 08 13:18:54.031118 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031100    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 13:18:54.031341 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031324    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 08 13:18:54.031468 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031427    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 08 13:18:54.031570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031485    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-scjzv\""
Sep 08 13:18:54.032086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031403    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 08 13:18:54.032086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031430    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 08 13:18:54.032086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031623    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 13:18:54.032086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031716    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 08 13:18:54.032086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031761    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 08 13:18:54.032086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031388    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 13:18:54.032086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.031738    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f5dzn\""
Sep 08 13:18:54.035742 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.035721    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 08 13:18:54.036446 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.036425    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 08 13:18:54.040284 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.040256    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8rdfw"]
Sep 08 13:18:54.041203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.041183    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 08 13:18:54.043380 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.043361    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-6lrnr"]
Sep 08 13:18:54.057130 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.057104    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-567cf589fd-rkshv"]
Sep 08 13:18:54.112687 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112651    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da748d4c-3623-4746-b899-479c9aaa81c1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.112687 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112686    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jv7s\" (UniqueName: \"kubernetes.io/projected/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-kube-api-access-6jv7s\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.112907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112711    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/da748d4c-3623-4746-b899-479c9aaa81c1-snapshots\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.112907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-stats-auth\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.112907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112765    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da748d4c-3623-4746-b899-479c9aaa81c1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.112907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112782    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-default-certificate\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.112907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112826    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da748d4c-3623-4746-b899-479c9aaa81c1-tmp\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.112907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112844    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da748d4c-3623-4746-b899-479c9aaa81c1-serving-cert\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.112907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptfzc\" (UniqueName: \"kubernetes.io/projected/574adb2c-aa3b-486f-bf43-06720a5f134f-kube-api-access-ptfzc\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.112907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112894    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.112907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112910    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/574adb2c-aa3b-486f-bf43-06720a5f134f-serving-cert\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.113238 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/574adb2c-aa3b-486f-bf43-06720a5f134f-trusted-ca\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.113238 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.112985    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.113238 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.113079    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ksb\" (UniqueName: \"kubernetes.io/projected/da748d4c-3623-4746-b899-479c9aaa81c1-kube-api-access-p2ksb\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.113238 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.113109    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/574adb2c-aa3b-486f-bf43-06720a5f134f-config\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.119215 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.119188    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"]
Sep 08 13:18:54.120948 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.120934    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.123756 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.123737    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 13:18:54.123756 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.123753    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 08 13:18:54.123756 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.123760    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z6p58\""
Sep 08 13:18:54.123998 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.123828    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 13:18:54.123998 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.123840    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 08 13:18:54.133541 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.133517    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"]
Sep 08 13:18:54.213458 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-default-certificate\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.213644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213469    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da748d4c-3623-4746-b899-479c9aaa81c1-tmp\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.213644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da748d4c-3623-4746-b899-479c9aaa81c1-serving-cert\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.213644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213506    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptfzc\" (UniqueName: \"kubernetes.io/projected/574adb2c-aa3b-486f-bf43-06720a5f134f-kube-api-access-ptfzc\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.213644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213523    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.213644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/574adb2c-aa3b-486f-bf43-06720a5f134f-serving-cert\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.213644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/574adb2c-aa3b-486f-bf43-06720a5f134f-trusted-ca\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.213644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213718    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213752    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8q6c\" (UniqueName: \"kubernetes.io/projected/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-kube-api-access-k8q6c\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.213770    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ksb\" (UniqueName: \"kubernetes.io/projected/da748d4c-3623-4746-b899-479c9aaa81c1-kube-api-access-p2ksb\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213846    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/574adb2c-aa3b-486f-bf43-06720a5f134f-config\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.213864    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.713841247 +0000 UTC m=+163.642744988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : secret "router-metrics-certs-default" not found
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213897    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213936    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da748d4c-3623-4746-b899-479c9aaa81c1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.213965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jv7s\" (UniqueName: \"kubernetes.io/projected/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-kube-api-access-6jv7s\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.214042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.214002    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/da748d4c-3623-4746-b899-479c9aaa81c1-snapshots\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.214523 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.214058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-stats-auth\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.214523 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.214093    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da748d4c-3623-4746-b899-479c9aaa81c1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.214523 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.214153    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/da748d4c-3623-4746-b899-479c9aaa81c1-tmp\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.214523 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.214285    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.714265729 +0000 UTC m=+163.643169452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:54.214736 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.214527    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/574adb2c-aa3b-486f-bf43-06720a5f134f-config\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.214736 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.214686    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/574adb2c-aa3b-486f-bf43-06720a5f134f-trusted-ca\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.214990 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.214923    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da748d4c-3623-4746-b899-479c9aaa81c1-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.215164 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.215120    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/da748d4c-3623-4746-b899-479c9aaa81c1-snapshots\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.215313 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.215293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da748d4c-3623-4746-b899-479c9aaa81c1-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.216695 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.216675    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/da748d4c-3623-4746-b899-479c9aaa81c1-serving-cert\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.216781 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.216767    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-stats-auth\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.216970 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.216947    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-default-certificate\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.217173 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.217157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/574adb2c-aa3b-486f-bf43-06720a5f134f-serving-cert\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.228834 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.228805    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jv7s\" (UniqueName: \"kubernetes.io/projected/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-kube-api-access-6jv7s\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.228955 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.228847    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptfzc\" (UniqueName: \"kubernetes.io/projected/574adb2c-aa3b-486f-bf43-06720a5f134f-kube-api-access-ptfzc\") pod \"console-operator-85b454874b-8rdfw\" (UID: \"574adb2c-aa3b-486f-bf43-06720a5f134f\") " pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.229098 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.229076    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ksb\" (UniqueName: \"kubernetes.io/projected/da748d4c-3623-4746-b899-479c9aaa81c1-kube-api-access-p2ksb\") pod \"insights-operator-76bcd6ccf5-6lrnr\" (UID: \"da748d4c-3623-4746-b899-479c9aaa81c1\") " pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.312703 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.312673    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:54.315427 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.315396    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.315515 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.315459    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8q6c\" (UniqueName: \"kubernetes.io/projected/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-kube-api-access-k8q6c\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.315515 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.315509    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.315597 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.315547    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:54.315639 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.315622    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls podName:ba4b3c60-a06b-4af3-9d0a-e80226cae5ab nodeName:}" failed. No retries permitted until 2026-09-08 13:18:54.815600397 +0000 UTC m=+163.744504204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-s657h" (UID: "ba4b3c60-a06b-4af3-9d0a-e80226cae5ab") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:54.316209 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.316192    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.326050 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.326028    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8q6c\" (UniqueName: \"kubernetes.io/projected/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-kube-api-access-k8q6c\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.334919 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.334892    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:18:54.347753 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.347731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr"
Sep 08 13:18:54.411222 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.411194    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:54.607455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.607382    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-8rdfw"]
Sep 08 13:18:54.610710 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:18:54.610682    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod574adb2c_aa3b_486f_bf43_06720a5f134f.slice/crio-5c2524c6405ce0f807e2e268eba4acf575bd620b496d9412687ef3e9dd2bab55 WatchSource:0}: Error finding container 5c2524c6405ce0f807e2e268eba4acf575bd620b496d9412687ef3e9dd2bab55: Status 404 returned error can't find the container with id 5c2524c6405ce0f807e2e268eba4acf575bd620b496d9412687ef3e9dd2bab55
Sep 08 13:18:54.626162 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.626127    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-6lrnr"]
Sep 08 13:18:54.629289 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:18:54.629263    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda748d4c_3623_4746_b899_479c9aaa81c1.slice/crio-76042a959d3aed96f707028417556bf253fe93cbc85a35e618b78a3b9f6840ab WatchSource:0}: Error finding container 76042a959d3aed96f707028417556bf253fe93cbc85a35e618b78a3b9f6840ab: Status 404 returned error can't find the container with id 76042a959d3aed96f707028417556bf253fe93cbc85a35e618b78a3b9f6840ab
Sep 08 13:18:54.718818 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.718776    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.718988 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.718838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:54.718988 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.718930    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:54.719080 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.719000    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:55.718981631 +0000 UTC m=+164.647885353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : secret "router-metrics-certs-default" not found
Sep 08 13:18:54.719080 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.719035    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:55.719027372 +0000 UTC m=+164.647931092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:54.819709 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.819666    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:54.819883 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.819730    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") pod \"image-registry-7789bf9865-vzcqq\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") " pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:18:54.819883 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.819824    2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 13:18:54.819883 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.819828    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:54.819883 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.819837    2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7789bf9865-vzcqq: secret "image-registry-tls" not found
Sep 08 13:18:54.820037 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.819901    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls podName:ba4b3c60-a06b-4af3-9d0a-e80226cae5ab nodeName:}" failed. No retries permitted until 2026-09-08 13:18:55.819880501 +0000 UTC m=+164.748784221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-s657h" (UID: "ba4b3c60-a06b-4af3-9d0a-e80226cae5ab") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:54.820037 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.819923    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls podName:12f11884-1f2b-49bf-99b4-4c7f5f299fbe nodeName:}" failed. No retries permitted until 2026-09-08 13:20:56.819913716 +0000 UTC m=+285.748817436 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls") pod "image-registry-7789bf9865-vzcqq" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe") : secret "image-registry-tls" not found
Sep 08 13:18:54.920502 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:54.920411    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:18:54.920647 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.920563    2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 13:18:54.920647 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:54.920637    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls podName:8e6529bc-e29e-406f-ae0e-55c85d81465b nodeName:}" failed. No retries permitted until 2026-09-08 13:20:56.92061423 +0000 UTC m=+285.849517950 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls") pod "dns-default-k6zh9" (UID: "8e6529bc-e29e-406f-ae0e-55c85d81465b") : secret "dns-default-metrics-tls" not found
Sep 08 13:18:55.106938 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:55.106897    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr" event={"ID":"da748d4c-3623-4746-b899-479c9aaa81c1","Type":"ContainerStarted","Data":"76042a959d3aed96f707028417556bf253fe93cbc85a35e618b78a3b9f6840ab"}
Sep 08 13:18:55.108260 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:55.108236    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" event={"ID":"574adb2c-aa3b-486f-bf43-06720a5f134f","Type":"ContainerStarted","Data":"5c2524c6405ce0f807e2e268eba4acf575bd620b496d9412687ef3e9dd2bab55"}
Sep 08 13:18:55.728588 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:55.728550    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:55.728777 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:55.728628    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:55.728777 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:55.728718    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:55.728942 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:55.728816    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:57.728800375 +0000 UTC m=+166.657704094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : secret "router-metrics-certs-default" not found
Sep 08 13:18:55.728942 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:55.728832    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:57.728824888 +0000 UTC m=+166.657728607 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:55.829252 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:55.829205    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:55.829444 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:55.829382    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:55.829511 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:55.829465    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls podName:ba4b3c60-a06b-4af3-9d0a-e80226cae5ab nodeName:}" failed. No retries permitted until 2026-09-08 13:18:57.829443708 +0000 UTC m=+166.758347612 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-s657h" (UID: "ba4b3c60-a06b-4af3-9d0a-e80226cae5ab") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:57.657603 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.657567    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rpgqv"]
Sep 08 13:18:57.659699 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.659673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.661585 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.661565    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 13:18:57.662120 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.662098    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kpczc\""
Sep 08 13:18:57.662366 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.662353    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 13:18:57.673424 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.673391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rpgqv"]
Sep 08 13:18:57.685310 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.685282    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:57.745559 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.745508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:57.745740 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.745604    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12dbfc49-1b02-4e95-9c56-91a93b3054f4-data-volume\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.745740 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.745639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:18:57.745740 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.745668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12dbfc49-1b02-4e95-9c56-91a93b3054f4-crio-socket\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.745740 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.745715    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.745740 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:57.745669    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:18:57.745977 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.745744    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz5vb\" (UniqueName: \"kubernetes.io/projected/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-api-access-qz5vb\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.745977 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:57.745785    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.74576928 +0000 UTC m=+170.674672999 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : secret "router-metrics-certs-default" not found
Sep 08 13:18:57.745977 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:57.745810    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.745795314 +0000 UTC m=+170.674699034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : configmap references non-existent config key: service-ca.crt
Sep 08 13:18:57.745977 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.745856    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.765668 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.765637    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:57.846983 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.846950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12dbfc49-1b02-4e95-9c56-91a93b3054f4-data-volume\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.847211 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.847028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12dbfc49-1b02-4e95-9c56-91a93b3054f4-crio-socket\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.847211 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.847076    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.847211 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.847102    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz5vb\" (UniqueName: \"kubernetes.io/projected/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-api-access-qz5vb\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.847211 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.847130    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.847211 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.847189    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/12dbfc49-1b02-4e95-9c56-91a93b3054f4-crio-socket\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.847446 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.847217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:18:57.847446 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:57.847237    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:57.847446 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:57.847266    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:57.847446 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:57.847307    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:58.347287094 +0000 UTC m=+167.276190815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:57.847446 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:57.847328    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:58.347317983 +0000 UTC m=+167.276221705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:57.847446 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:57.847327    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:57.847446 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.847368    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/12dbfc49-1b02-4e95-9c56-91a93b3054f4-data-volume\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.847446 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:57.847376    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls podName:ba4b3c60-a06b-4af3-9d0a-e80226cae5ab nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.847361157 +0000 UTC m=+170.776264891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-s657h" (UID: "ba4b3c60-a06b-4af3-9d0a-e80226cae5ab") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:18:57.857241 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.857213    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz5vb\" (UniqueName: \"kubernetes.io/projected/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-api-access-qz5vb\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:57.965323 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:57.965240    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:58.123991 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.123952    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr" event={"ID":"da748d4c-3623-4746-b899-479c9aaa81c1","Type":"ContainerStarted","Data":"ba24ec77905dc35e873b24d211a019cbdb3b8d5d3ad2798b469aed605807b325"}
Sep 08 13:18:58.125618 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.125596    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/0.log"
Sep 08 13:18:58.125743 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.125632    2583 generic.go:356] "Generic (PLEG): container finished" podID="574adb2c-aa3b-486f-bf43-06720a5f134f" containerID="431d35c8380290a0a7557af62d2d20589f9a6b8ad63054b9cdbdf42111eba410" exitCode=255
Sep 08 13:18:58.125743 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.125702    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" event={"ID":"574adb2c-aa3b-486f-bf43-06720a5f134f","Type":"ContainerDied","Data":"431d35c8380290a0a7557af62d2d20589f9a6b8ad63054b9cdbdf42111eba410"}
Sep 08 13:18:58.125910 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.125898    2583 scope.go:117] "RemoveContainer" containerID="431d35c8380290a0a7557af62d2d20589f9a6b8ad63054b9cdbdf42111eba410"
Sep 08 13:18:58.142159 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.141813    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr" podStartSLOduration=2.67652066 podStartE2EDuration="5.14179103s" totalImagesPullingTime="2.46527037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:18:58.140799506 +0000 UTC m=+167.069703268" watchObservedRunningTime="2026-09-08 13:18:58.14179103 +0000 UTC m=+167.070694775"
Sep 08 13:18:58.351998 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.351956    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:58.352227 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.352028    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:58.352227 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:58.352108    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:58.352227 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:58.352173    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:59.352157974 +0000 UTC m=+168.281061694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:58.352227 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:58.352108    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:58.352396 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:58.352262    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:18:59.352248256 +0000 UTC m=+168.281151980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:58.819344 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.819296    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:58.962703 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:58.962671    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:18:59.131032 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:59.130938    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/1.log"
Sep 08 13:18:59.131381 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:59.131367    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/0.log"
Sep 08 13:18:59.131428 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:59.131401    2583 generic.go:356] "Generic (PLEG): container finished" podID="574adb2c-aa3b-486f-bf43-06720a5f134f" containerID="9c79746a76cb0750b9ffaebaf01d45aabc9f42ffc4a8d94972fb73d4abf6f067" exitCode=255
Sep 08 13:18:59.131500 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:59.131482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" event={"ID":"574adb2c-aa3b-486f-bf43-06720a5f134f","Type":"ContainerDied","Data":"9c79746a76cb0750b9ffaebaf01d45aabc9f42ffc4a8d94972fb73d4abf6f067"}
Sep 08 13:18:59.131548 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:59.131536    2583 scope.go:117] "RemoveContainer" containerID="431d35c8380290a0a7557af62d2d20589f9a6b8ad63054b9cdbdf42111eba410"
Sep 08 13:18:59.131766 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:59.131748    2583 scope.go:117] "RemoveContainer" containerID="9c79746a76cb0750b9ffaebaf01d45aabc9f42ffc4a8d94972fb73d4abf6f067"
Sep 08 13:18:59.131929 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:59.131912    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8rdfw_openshift-console-operator(574adb2c-aa3b-486f-bf43-06720a5f134f)\"" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" podUID="574adb2c-aa3b-486f-bf43-06720a5f134f"
Sep 08 13:18:59.364613 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:59.364570    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:59.364613 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:59.364621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:18:59.364932 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:59.364703    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:18:59.364932 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:59.364761    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:59.364932 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:59.364806    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.364782343 +0000 UTC m=+170.293686062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : configmap "kube-rbac-proxy" not found
Sep 08 13:18:59.364932 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:18:59.364829    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:01.364819259 +0000 UTC m=+170.293722978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:18:59.639287 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:18:59.639262    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9frn_02703e20-c856-4652-9e02-6086e7b11179/dns-node-resolver/0.log"
Sep 08 13:19:00.135240 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:00.135213    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/1.log"
Sep 08 13:19:00.135613 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:00.135548    2583 scope.go:117] "RemoveContainer" containerID="9c79746a76cb0750b9ffaebaf01d45aabc9f42ffc4a8d94972fb73d4abf6f067"
Sep 08 13:19:00.135714 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:00.135694    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8rdfw_openshift-console-operator(574adb2c-aa3b-486f-bf43-06720a5f134f)\"" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" podUID="574adb2c-aa3b-486f-bf43-06720a5f134f"
Sep 08 13:19:00.841894 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:00.841860    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vrfh7_817efad1-a9dc-4e98-ae2d-396f97d0eda9/node-ca/0.log"
Sep 08 13:19:01.126200 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:01.126117    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:01.190495 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:01.190463    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:01.381615 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:01.381529    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:01.381615 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:01.381573    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:01.381794 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:01.381652    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:01.381794 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:01.381702    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:05.381688508 +0000 UTC m=+174.310592229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:01.381794 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:01.381707    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:01.381794 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:01.381763    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:05.381745388 +0000 UTC m=+174.310649108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:01.590766 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:01.590728    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:19:01.639595 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:01.639488    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9qg7h_a352c5b0-060c-4938-9775-9138f5461a28/serve-healthcheck-canary/0.log"
Sep 08 13:19:01.784870 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:01.784826    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:01.785064 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:01.784898    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:01.785064 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:01.784986    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:19:01.785064 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:01.785064    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.785046978 +0000 UTC m=+178.713950719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : secret "router-metrics-certs-default" not found
Sep 08 13:19:01.785184 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:01.785085    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.785070088 +0000 UTC m=+178.713973837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : configmap references non-existent config key: service-ca.crt
Sep 08 13:19:01.885380 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:01.885341    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:19:01.885576 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:01.885502    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:01.885629 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:01.885588    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls podName:ba4b3c60-a06b-4af3-9d0a-e80226cae5ab nodeName:}" failed. No retries permitted until 2026-09-08 13:19:09.885565595 +0000 UTC m=+178.814469334 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-s657h" (UID: "ba4b3c60-a06b-4af3-9d0a-e80226cae5ab") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:02.151701 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:02.151671    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:02.226143 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:02.226116    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:02.460203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:02.460096    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:02.568555 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:02.568517    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:04.335222 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.335188    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:19:04.335630 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.335256    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:19:04.335630 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.335584    2583 scope.go:117] "RemoveContainer" containerID="9c79746a76cb0750b9ffaebaf01d45aabc9f42ffc4a8d94972fb73d4abf6f067"
Sep 08 13:19:04.335743 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:04.335724    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-8rdfw_openshift-console-operator(574adb2c-aa3b-486f-bf43-06720a5f134f)\"" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" podUID="574adb2c-aa3b-486f-bf43-06720a5f134f"
Sep 08 13:19:04.745970 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.745891    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7brz2"]
Sep 08 13:19:04.748701 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.748686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:04.750821 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.750801    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 13:19:04.750960 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.750800    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 13:19:04.751275 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.751261    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lc5pj\""
Sep 08 13:19:04.751332 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.751287    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 13:19:04.751426 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.751407    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 13:19:04.757768 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.757750    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7brz2"]
Sep 08 13:19:04.911381 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.911345    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxhjc\" (UniqueName: \"kubernetes.io/projected/97ae32a1-7210-4473-963d-4530572ffef9-kube-api-access-hxhjc\") pod \"service-ca-75d99c484-7brz2\" (UID: \"97ae32a1-7210-4473-963d-4530572ffef9\") " pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:04.911581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.911444    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97ae32a1-7210-4473-963d-4530572ffef9-signing-cabundle\") pod \"service-ca-75d99c484-7brz2\" (UID: \"97ae32a1-7210-4473-963d-4530572ffef9\") " pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:04.911581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:04.911530    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97ae32a1-7210-4473-963d-4530572ffef9-signing-key\") pod \"service-ca-75d99c484-7brz2\" (UID: \"97ae32a1-7210-4473-963d-4530572ffef9\") " pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:05.012648 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.012556    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97ae32a1-7210-4473-963d-4530572ffef9-signing-cabundle\") pod \"service-ca-75d99c484-7brz2\" (UID: \"97ae32a1-7210-4473-963d-4530572ffef9\") " pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:05.012648 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.012626    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97ae32a1-7210-4473-963d-4530572ffef9-signing-key\") pod \"service-ca-75d99c484-7brz2\" (UID: \"97ae32a1-7210-4473-963d-4530572ffef9\") " pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:05.012875 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.012672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxhjc\" (UniqueName: \"kubernetes.io/projected/97ae32a1-7210-4473-963d-4530572ffef9-kube-api-access-hxhjc\") pod \"service-ca-75d99c484-7brz2\" (UID: \"97ae32a1-7210-4473-963d-4530572ffef9\") " pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:05.013315 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.013292    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/97ae32a1-7210-4473-963d-4530572ffef9-signing-cabundle\") pod \"service-ca-75d99c484-7brz2\" (UID: \"97ae32a1-7210-4473-963d-4530572ffef9\") " pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:05.015195 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.015174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/97ae32a1-7210-4473-963d-4530572ffef9-signing-key\") pod \"service-ca-75d99c484-7brz2\" (UID: \"97ae32a1-7210-4473-963d-4530572ffef9\") " pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:05.020543 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.020517    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxhjc\" (UniqueName: \"kubernetes.io/projected/97ae32a1-7210-4473-963d-4530572ffef9-kube-api-access-hxhjc\") pod \"service-ca-75d99c484-7brz2\" (UID: \"97ae32a1-7210-4473-963d-4530572ffef9\") " pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:05.057552 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.057516    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-7brz2"
Sep 08 13:19:05.332461 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.332425    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-7brz2"]
Sep 08 13:19:05.335587 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:19:05.335559    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ae32a1_7210_4473_963d_4530572ffef9.slice/crio-8c4da70771fa872d0bd6389941a145e649bcb08e257606ad6b5a55969b7e1797 WatchSource:0}: Error finding container 8c4da70771fa872d0bd6389941a145e649bcb08e257606ad6b5a55969b7e1797: Status 404 returned error can't find the container with id 8c4da70771fa872d0bd6389941a145e649bcb08e257606ad6b5a55969b7e1797
Sep 08 13:19:05.416573 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.416543    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:05.416735 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:05.416583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:05.416735 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:05.416660    2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 13:19:05.416735 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:05.416707    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:05.416847 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:05.416716    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:13.416702264 +0000 UTC m=+182.345605985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : configmap "kube-rbac-proxy" not found
Sep 08 13:19:05.416847 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:05.416786    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:13.416766284 +0000 UTC m=+182.345670013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:06.152863 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:06.152813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-7brz2" event={"ID":"97ae32a1-7210-4473-963d-4530572ffef9","Type":"ContainerStarted","Data":"8c4da70771fa872d0bd6389941a145e649bcb08e257606ad6b5a55969b7e1797"}
Sep 08 13:19:06.386208 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:06.386162    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:07.156865 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:07.156827    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-7brz2" event={"ID":"97ae32a1-7210-4473-963d-4530572ffef9","Type":"ContainerStarted","Data":"1781a97658e9ba97ebb3102f9b9f680a059a84131f69a76271cde52379201558"}
Sep 08 13:19:07.172925 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:07.172867    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-7brz2" podStartSLOduration=1.506325468 podStartE2EDuration="3.172849535s" totalImagesPullingTime="1.666524067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:07.172141032 +0000 UTC m=+176.101044799" watchObservedRunningTime="2026-09-08 13:19:07.172849535 +0000 UTC m=+176.101753276"
Sep 08 13:19:09.856668 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:09.856621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:09.857089 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:09.856689    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:09.857089 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:09.856784    2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 13:19:09.857089 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:09.856865    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:25.856843556 +0000 UTC m=+194.785747300 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : secret "router-metrics-certs-default" not found
Sep 08 13:19:09.857089 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:09.856881    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle podName:dabbf1a4-723b-4ccc-b631-ab3db3d1ff05 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:25.856875719 +0000 UTC m=+194.785779438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle") pod "router-default-567cf589fd-rkshv" (UID: "dabbf1a4-723b-4ccc-b631-ab3db3d1ff05") : configmap references non-existent config key: service-ca.crt
Sep 08 13:19:09.958175 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:09.958143    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:19:09.958356 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:09.958251    2583 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:09.958356 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:09.958312    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls podName:ba4b3c60-a06b-4af3-9d0a-e80226cae5ab nodeName:}" failed. No retries permitted until 2026-09-08 13:19:25.958299481 +0000 UTC m=+194.887203201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-s657h" (UID: "ba4b3c60-a06b-4af3-9d0a-e80226cae5ab") : secret "cluster-monitoring-operator-tls" not found
Sep 08 13:19:10.255078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:10.254986    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:10.332171 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:10.332140    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:10.538493 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:10.538409    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:10.704062 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:10.704036    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:10.904652 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:10.904574    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:11.112834 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:11.112803    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:11.176380 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:11.176297    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:11.543732 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:11.543644    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:11.729920 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:11.729889    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.322589 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:12.322558    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.391956 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:12.391922    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:12.726226 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:12.726143    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:13.111751 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:13.111713    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:13.317833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:13.317794    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:13.492342 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:13.492264    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:13.492342 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:13.492322    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:13.492739 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:13.492384    2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:13.492739 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:13.492455    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls podName:12dbfc49-1b02-4e95-9c56-91a93b3054f4 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:29.492439022 +0000 UTC m=+198.421342743 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rpgqv" (UID: "12dbfc49-1b02-4e95-9c56-91a93b3054f4") : secret "insights-runtime-extractor-tls" not found
Sep 08 13:19:13.492902 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:13.492882    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/12dbfc49-1b02-4e95-9c56-91a93b3054f4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:13.718877 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:13.718842    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.136698 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:14.136661    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.236867 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:14.236836    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:14.337188 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:14.337154    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:16.590464 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:16.590437    2583 scope.go:117] "RemoveContainer" containerID="9c79746a76cb0750b9ffaebaf01d45aabc9f42ffc4a8d94972fb73d4abf6f067"
Sep 08 13:19:17.196959 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:17.196927    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:19:17.197365 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:17.197348    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/1.log"
Sep 08 13:19:17.197427 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:17.197386    2583 generic.go:356] "Generic (PLEG): container finished" podID="574adb2c-aa3b-486f-bf43-06720a5f134f" containerID="cf9f8694ad5a80305c3005abf96f85d3ed4456797e75d8a38a618474ff25e8a6" exitCode=255
Sep 08 13:19:17.197478 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:17.197433    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" event={"ID":"574adb2c-aa3b-486f-bf43-06720a5f134f","Type":"ContainerDied","Data":"cf9f8694ad5a80305c3005abf96f85d3ed4456797e75d8a38a618474ff25e8a6"}
Sep 08 13:19:17.197528 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:17.197505    2583 scope.go:117] "RemoveContainer" containerID="9c79746a76cb0750b9ffaebaf01d45aabc9f42ffc4a8d94972fb73d4abf6f067"
Sep 08 13:19:17.197884 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:17.197863    2583 scope.go:117] "RemoveContainer" containerID="cf9f8694ad5a80305c3005abf96f85d3ed4456797e75d8a38a618474ff25e8a6"
Sep 08 13:19:17.198045 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:17.198026    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8rdfw_openshift-console-operator(574adb2c-aa3b-486f-bf43-06720a5f134f)\"" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" podUID="574adb2c-aa3b-486f-bf43-06720a5f134f"
Sep 08 13:19:17.990044 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:17.989997    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:18.051586 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:18.051560    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:18.202183 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:18.202151    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:19:18.517215 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:18.517173    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:18.724589 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:18.724558    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:18.919076 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:18.919047    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:19.125854 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:19.125824    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:24.335539 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:24.335485    2583 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:19:24.335539 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:24.335558    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:19:24.336035 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:24.335865    2583 scope.go:117] "RemoveContainer" containerID="cf9f8694ad5a80305c3005abf96f85d3ed4456797e75d8a38a618474ff25e8a6"
Sep 08 13:19:24.336077 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:24.336036    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-8rdfw_openshift-console-operator(574adb2c-aa3b-486f-bf43-06720a5f134f)\"" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" podUID="574adb2c-aa3b-486f-bf43-06720a5f134f"
Sep 08 13:19:25.894567 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:25.894526    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:25.894926 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:25.894588    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:25.895204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:25.895184    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-service-ca-bundle\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:25.897005 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:25.896978    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dabbf1a4-723b-4ccc-b631-ab3db3d1ff05-metrics-certs\") pod \"router-default-567cf589fd-rkshv\" (UID: \"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05\") " pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:25.995374 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:25.995315    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:19:25.997788 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:25.997768    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba4b3c60-a06b-4af3-9d0a-e80226cae5ab-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-s657h\" (UID: \"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:19:26.154954 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:26.154876    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-scjzv\""
Sep 08 13:19:26.163787 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:26.163769    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:26.231715 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:26.231682    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-z6p58\""
Sep 08 13:19:26.240450 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:26.240430    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"
Sep 08 13:19:26.332520 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:26.332488    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:26.444804 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:26.444715    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-567cf589fd-rkshv"]
Sep 08 13:19:26.448356 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:19:26.448323    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddabbf1a4_723b_4ccc_b631_ab3db3d1ff05.slice/crio-d53a30448308b32e74f62d75cf0a824a6464a67c1252839ca52ba3dd0835eaa0 WatchSource:0}: Error finding container d53a30448308b32e74f62d75cf0a824a6464a67c1252839ca52ba3dd0835eaa0: Status 404 returned error can't find the container with id d53a30448308b32e74f62d75cf0a824a6464a67c1252839ca52ba3dd0835eaa0
Sep 08 13:19:26.453393 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:26.453367    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:26.511742 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:26.511714    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h"]
Sep 08 13:19:26.514821 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:19:26.514790    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4b3c60_a06b_4af3_9d0a_e80226cae5ab.slice/crio-3b07cf295ed1196f115657de2f69763032923d19a670d81cbd81260cda60da42 WatchSource:0}: Error finding container 3b07cf295ed1196f115657de2f69763032923d19a670d81cbd81260cda60da42: Status 404 returned error can't find the container with id 3b07cf295ed1196f115657de2f69763032923d19a670d81cbd81260cda60da42
Sep 08 13:19:27.226900 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:27.226865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h" event={"ID":"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab","Type":"ContainerStarted","Data":"3b07cf295ed1196f115657de2f69763032923d19a670d81cbd81260cda60da42"}
Sep 08 13:19:27.230143 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:27.230116    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-567cf589fd-rkshv" event={"ID":"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05","Type":"ContainerStarted","Data":"d32e57a2c293df25386630f16e9aae285ee671316a7eb306f013f9de9d257078"}
Sep 08 13:19:27.230275 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:27.230157    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-567cf589fd-rkshv" event={"ID":"dabbf1a4-723b-4ccc-b631-ab3db3d1ff05","Type":"ContainerStarted","Data":"d53a30448308b32e74f62d75cf0a824a6464a67c1252839ca52ba3dd0835eaa0"}
Sep 08 13:19:27.250510 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:27.250463    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-567cf589fd-rkshv" podStartSLOduration=34.250447627 podStartE2EDuration="34.250447627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:27.250088298 +0000 UTC m=+196.178992050" watchObservedRunningTime="2026-09-08 13:19:27.250447627 +0000 UTC m=+196.179351369"
Sep 08 13:19:28.163943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:28.163909    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:28.166469 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:28.166449    2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:28.232445 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:28.232418    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:28.233385 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:28.233365    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:28.234486 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:28.234467    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-567cf589fd-rkshv"
Sep 08 13:19:28.333650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:28.333618    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:28.411336 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:28.411298    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:29.320243 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:29.320209    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:29.494399 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:29.494367    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:29.522369 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:29.522342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:29.524733 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:29.524708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/12dbfc49-1b02-4e95-9c56-91a93b3054f4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rpgqv\" (UID: \"12dbfc49-1b02-4e95-9c56-91a93b3054f4\") " pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:29.770908 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:29.770827    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kpczc\""
Sep 08 13:19:29.779466 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:29.779431    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rpgqv"
Sep 08 13:19:30.055647 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:30.055573    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rpgqv"]
Sep 08 13:19:30.059193 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:19:30.059166    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12dbfc49_1b02_4e95_9c56_91a93b3054f4.slice/crio-8edabe9224fd1524581d6c1ee205dce1132bb040e78e3c15feeeed22dfafe73d WatchSource:0}: Error finding container 8edabe9224fd1524581d6c1ee205dce1132bb040e78e3c15feeeed22dfafe73d: Status 404 returned error can't find the container with id 8edabe9224fd1524581d6c1ee205dce1132bb040e78e3c15feeeed22dfafe73d
Sep 08 13:19:30.241917 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:30.241887    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpgqv" event={"ID":"12dbfc49-1b02-4e95-9c56-91a93b3054f4","Type":"ContainerStarted","Data":"cc3a210c4d5b73d0900a3c9bf9d4467e306b240e28f4daa8a46c581ca971ab57"}
Sep 08 13:19:30.242115 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:30.241948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpgqv" event={"ID":"12dbfc49-1b02-4e95-9c56-91a93b3054f4","Type":"ContainerStarted","Data":"8edabe9224fd1524581d6c1ee205dce1132bb040e78e3c15feeeed22dfafe73d"}
Sep 08 13:19:31.640178 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:31.640081    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:31.742322 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:31.742291    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:31.822440 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:31.822406    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:32.089233 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:32.089204    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:32.249907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:32.249874    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h" event={"ID":"ba4b3c60-a06b-4af3-9d0a-e80226cae5ab","Type":"ContainerStarted","Data":"40f33bebd08f0cbaebb4a092dcab7c4dd56434ea9c77c73505ef2325521b32aa"}
Sep 08 13:19:32.269206 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:32.269163    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-s657h" podStartSLOduration=33.461692236 podStartE2EDuration="38.269147848s" totalImagesPullingTime="4.807455612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:32.268243213 +0000 UTC m=+201.197146980" watchObservedRunningTime="2026-09-08 13:19:32.269147848 +0000 UTC m=+201.198051591"
Sep 08 13:19:34.263983 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:34.263949    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpgqv" event={"ID":"12dbfc49-1b02-4e95-9c56-91a93b3054f4","Type":"ContainerStarted","Data":"a3d2c2ebb2206298b2f1f4f5765a1155e23fa58f7c3dc7e2f2b3160f44d8c4b2"}
Sep 08 13:19:35.847939 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.847906    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:35.889675 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.889648    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"]
Sep 08 13:19:35.892701 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.892686    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:35.895239 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.895195    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 08 13:19:35.895350 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.895206    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-897lm\""
Sep 08 13:19:35.895763 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.895743    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 08 13:19:35.895935 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.895920    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 13:19:35.902087 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.902067    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"]
Sep 08 13:19:35.925737 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.925712    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:35.974991 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.974962    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:35.975144 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.974997    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:35.975144 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.975041    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:35.975144 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:35.975119    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vgx\" (UniqueName: \"kubernetes.io/projected/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-kube-api-access-h6vgx\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.076567 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.076538    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.076719 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.076574    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.076719 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.076605    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.076719 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.076643    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vgx\" (UniqueName: \"kubernetes.io/projected/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-kube-api-access-h6vgx\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.077255 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.077227    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.079310 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.079280    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.079310 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.079297    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.086224 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.086197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vgx\" (UniqueName: \"kubernetes.io/projected/65090ac4-f95f-42ba-8c98-a8455ab7e7c3-kube-api-access-h6vgx\") pod \"prometheus-operator-6f4946db56-2pf9c\" (UID: \"65090ac4-f95f-42ba-8c98-a8455ab7e7c3\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.202131 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.202057    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"
Sep 08 13:19:36.273003 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.272961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rpgqv" event={"ID":"12dbfc49-1b02-4e95-9c56-91a93b3054f4","Type":"ContainerStarted","Data":"7b1b39082f57a9173da5410fd28f241d3068c2625d036218d7ad47315217496f"}
Sep 08 13:19:36.292966 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.292916    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rpgqv" podStartSLOduration=34.145223484 podStartE2EDuration="39.292900471s" totalImagesPullingTime="5.147676987s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 13:18:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:36.292696443 +0000 UTC m=+205.221600207" watchObservedRunningTime="2026-09-08 13:19:36.292900471 +0000 UTC m=+205.221804234"
Sep 08 13:19:36.472184 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:36.472108    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-2pf9c"]
Sep 08 13:19:36.475462 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:19:36.475428    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65090ac4_f95f_42ba_8c98_a8455ab7e7c3.slice/crio-414f991dcae86e39f7d702929f85c1f78edbe064c8d6dfae2436afac7eecc2c1 WatchSource:0}: Error finding container 414f991dcae86e39f7d702929f85c1f78edbe064c8d6dfae2436afac7eecc2c1: Status 404 returned error can't find the container with id 414f991dcae86e39f7d702929f85c1f78edbe064c8d6dfae2436afac7eecc2c1
Sep 08 13:19:37.178545 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:37.178517    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:37.277890 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:37.277858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c" event={"ID":"65090ac4-f95f-42ba-8c98-a8455ab7e7c3","Type":"ContainerStarted","Data":"414f991dcae86e39f7d702929f85c1f78edbe064c8d6dfae2436afac7eecc2c1"}
Sep 08 13:19:37.589840 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:37.589807    2583 scope.go:117] "RemoveContainer" containerID="cf9f8694ad5a80305c3005abf96f85d3ed4456797e75d8a38a618474ff25e8a6"
Sep 08 13:19:38.284126 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.284100    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:19:38.284526 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.284177    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" event={"ID":"574adb2c-aa3b-486f-bf43-06720a5f134f","Type":"ContainerStarted","Data":"027820c5e464fced9560d8eb239211cad1720ce1d9a03a6f733d4b8a0d0af472"}
Sep 08 13:19:38.284526 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.284447    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:19:38.302787 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.302725    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-8rdfw" podStartSLOduration=42.817123347 podStartE2EDuration="45.302705246s" totalImagesPullingTime="2.485581899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:18:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:38.301255882 +0000 UTC m=+207.230159641" watchObservedRunningTime="2026-09-08 13:19:38.302705246 +0000 UTC m=+207.231608991"
Sep 08 13:19:38.759679 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.759653    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-8rdfw"
Sep 08 13:19:38.946672 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.946642    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-nxwsj"]
Sep 08 13:19:38.949694 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.949671    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-nxwsj"
Sep 08 13:19:38.951622 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.951551    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-wwgrf\""
Sep 08 13:19:38.951752 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.951732    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 13:19:38.952098 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.952076    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 13:19:38.961042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.960994    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-nxwsj"]
Sep 08 13:19:38.962039 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.961999    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:38.999158 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:38.999128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lgvd\" (UniqueName: \"kubernetes.io/projected/ec637e70-0655-4e0f-9da2-a4025cfd7e0f-kube-api-access-8lgvd\") pod \"downloads-85d98c998f-nxwsj\" (UID: \"ec637e70-0655-4e0f-9da2-a4025cfd7e0f\") " pod="openshift-console/downloads-85d98c998f-nxwsj"
Sep 08 13:19:39.061704 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.061628    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:39.100125 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.100091    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lgvd\" (UniqueName: \"kubernetes.io/projected/ec637e70-0655-4e0f-9da2-a4025cfd7e0f-kube-api-access-8lgvd\") pod \"downloads-85d98c998f-nxwsj\" (UID: \"ec637e70-0655-4e0f-9da2-a4025cfd7e0f\") " pod="openshift-console/downloads-85d98c998f-nxwsj"
Sep 08 13:19:39.108716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.108684    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lgvd\" (UniqueName: \"kubernetes.io/projected/ec637e70-0655-4e0f-9da2-a4025cfd7e0f-kube-api-access-8lgvd\") pod \"downloads-85d98c998f-nxwsj\" (UID: \"ec637e70-0655-4e0f-9da2-a4025cfd7e0f\") " pod="openshift-console/downloads-85d98c998f-nxwsj"
Sep 08 13:19:39.162727 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.162702    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:39.260765 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.260736    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-nxwsj"
Sep 08 13:19:39.261673 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.261648    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:39.289441 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.289413    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c" event={"ID":"65090ac4-f95f-42ba-8c98-a8455ab7e7c3","Type":"ContainerStarted","Data":"94a16f2a4ad9931cf91737ad6bd12afe1a28c7f1cd9c52c6e147e78fd2ec0000"}
Sep 08 13:19:39.289736 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.289462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c" event={"ID":"65090ac4-f95f-42ba-8c98-a8455ab7e7c3","Type":"ContainerStarted","Data":"e420474c242ea3708214ab066129638fdd4cc42e142be72301521c56e2091db0"}
Sep 08 13:19:39.431974 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.431943    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:39.534691 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.534639    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-2pf9c" podStartSLOduration=2.232326698 podStartE2EDuration="4.534619116s" totalImagesPullingTime="2.302292418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:39.310795144 +0000 UTC m=+208.239698901" watchObservedRunningTime="2026-09-08 13:19:39.534619116 +0000 UTC m=+208.463522857"
Sep 08 13:19:39.534873 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.534784    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-nxwsj"]
Sep 08 13:19:39.535906 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:39.535887    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:39.538106 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:19:39.538083    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec637e70_0655_4e0f_9da2_a4025cfd7e0f.slice/crio-14460904b7f7bfbbebd10ba8240c29ba0e3aa53f8fad0f61108343019751dfeb WatchSource:0}: Error finding container 14460904b7f7bfbbebd10ba8240c29ba0e3aa53f8fad0f61108343019751dfeb: Status 404 returned error can't find the container with id 14460904b7f7bfbbebd10ba8240c29ba0e3aa53f8fad0f61108343019751dfeb
Sep 08 13:19:40.293293 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:40.293253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-nxwsj" event={"ID":"ec637e70-0655-4e0f-9da2-a4025cfd7e0f","Type":"ContainerStarted","Data":"14460904b7f7bfbbebd10ba8240c29ba0e3aa53f8fad0f61108343019751dfeb"}
Sep 08 13:19:41.290626 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.289301    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"]
Sep 08 13:19:41.293357 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.292805    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.299665 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.298301    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-h7lqz\""
Sep 08 13:19:41.299665 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.298498    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 13:19:41.299665 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.298697    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 13:19:41.299665 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.298860    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 13:19:41.311258 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.311187    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"]
Sep 08 13:19:41.320605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.318722    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bnm9h"]
Sep 08 13:19:41.323479 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.322576    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.326469 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.326280    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4t6zk\""
Sep 08 13:19:41.326799 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.326625    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 13:19:41.327568 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.327211    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 13:19:41.327568 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.327430    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 13:19:41.421256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421229    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-wtmp\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.421397 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl7mf\" (UniqueName: \"kubernetes.io/projected/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-kube-api-access-hl7mf\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.421397 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421301    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.421397 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421382    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.421566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421448    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-tls\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.421566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421486    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-sys\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.421566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421511    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c689c\" (UniqueName: \"kubernetes.io/projected/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-api-access-c689c\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.421566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421548    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-metrics-client-ca\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.421765 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421616    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-root\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.421765 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-textfile\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.421765 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421731    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.421907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421761    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-accelerators-collector-config\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.421907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421789    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.421907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421814    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.421907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.421841    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522571    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-tls\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522618    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-sys\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522659    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c689c\" (UniqueName: \"kubernetes.io/projected/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-api-access-c689c\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522700    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-metrics-client-ca\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522738    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-root\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522750    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-sys\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522783    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-textfile\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522811    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522837    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-accelerators-collector-config\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522889    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522916    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522958    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-wtmp\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.522981    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl7mf\" (UniqueName: \"kubernetes.io/projected/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-kube-api-access-hl7mf\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.523035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.523070    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.523825 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:41.522704    2583 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:41.523233    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-tls podName:3ce5a7b7-4478-451d-b4fd-fc88df9f91de nodeName:}" failed. No retries permitted until 2026-09-08 13:19:42.023213048 +0000 UTC m=+210.952116773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-tls") pod "node-exporter-bnm9h" (UID: "3ce5a7b7-4478-451d-b4fd-fc88df9f91de") : secret "node-exporter-tls" not found
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.523258    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-wtmp\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.523395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-metrics-client-ca\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.523437    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-root\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.523444    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:41.523168    2583 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:41.523498    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-tls podName:90b4ca0f-0a3d-449c-86cf-0df165ac2ee8 nodeName:}" failed. No retries permitted until 2026-09-08 13:19:42.023485212 +0000 UTC m=+210.952388937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-mdxjp" (UID: "90b4ca0f-0a3d-449c-86cf-0df165ac2ee8") : secret "kube-state-metrics-tls" not found
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.523781    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-textfile\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.523915    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-accelerators-collector-config\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.524029    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.524816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.524754    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.526710 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.526667    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:41.529532 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.529490    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.535809 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.535764    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl7mf\" (UniqueName: \"kubernetes.io/projected/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-kube-api-access-hl7mf\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:41.537125 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:41.537085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c689c\" (UniqueName: \"kubernetes.io/projected/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-api-access-c689c\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:42.028370 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.028328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-tls\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:42.028557 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.028435    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:42.031344 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.031301    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ce5a7b7-4478-451d-b4fd-fc88df9f91de-node-exporter-tls\") pod \"node-exporter-bnm9h\" (UID: \"3ce5a7b7-4478-451d-b4fd-fc88df9f91de\") " pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:42.031485 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.031450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90b4ca0f-0a3d-449c-86cf-0df165ac2ee8-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-mdxjp\" (UID: \"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:42.209669 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.209626    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"
Sep 08 13:19:42.241173 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.241145    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bnm9h"
Sep 08 13:19:42.256533 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:19:42.256502    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce5a7b7_4478_451d_b4fd_fc88df9f91de.slice/crio-b2e8325cbbf8eb199d97ee8aab1bd64868116c2e1c13ee60f48a552853b8ff5c WatchSource:0}: Error finding container b2e8325cbbf8eb199d97ee8aab1bd64868116c2e1c13ee60f48a552853b8ff5c: Status 404 returned error can't find the container with id b2e8325cbbf8eb199d97ee8aab1bd64868116c2e1c13ee60f48a552853b8ff5c
Sep 08 13:19:42.310488 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.310402    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bnm9h" event={"ID":"3ce5a7b7-4478-451d-b4fd-fc88df9f91de","Type":"ContainerStarted","Data":"b2e8325cbbf8eb199d97ee8aab1bd64868116c2e1c13ee60f48a552853b8ff5c"}
Sep 08 13:19:42.370283 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.370252    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:19:42.375465 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.375442    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.377693 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.377556    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 13:19:42.381531 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.381509    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 13:19:42.382912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.382294    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 13:19:42.382912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.382480    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 13:19:42.382912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.382537    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 13:19:42.382912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.382655    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 13:19:42.382912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.382768    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 13:19:42.382912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.382837    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zhnnr\""
Sep 08 13:19:42.383348 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.382958    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 13:19:42.383348 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.383038    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 13:19:42.391965 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.391943    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:19:42.432283 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432247    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432300    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432329    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-web-config\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432433    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx7qg\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-kube-api-access-hx7qg\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432654 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432654 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432654 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432526    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432654 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432552    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432654 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432608    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-out\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432654 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432627    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432883 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432668    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.432883 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.432692    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.510659 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.510625    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp"]
Sep 08 13:19:42.514678 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:19:42.514647    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b4ca0f_0a3d_449c_86cf_0df165ac2ee8.slice/crio-bb86bc5b095b03fa40584afb9256be0935c9db7f1e2cc5dc83bf87508435a3aa WatchSource:0}: Error finding container bb86bc5b095b03fa40584afb9256be0935c9db7f1e2cc5dc83bf87508435a3aa: Status 404 returned error can't find the container with id bb86bc5b095b03fa40584afb9256be0935c9db7f1e2cc5dc83bf87508435a3aa
Sep 08 13:19:42.533674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533545    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx7qg\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-kube-api-access-hx7qg\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.533674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.533674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533638    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.533873 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533662    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.533873 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.533873 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533767    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-out\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.533873 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533804    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.533873 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533849    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.534143 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533875    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.534143 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.534143 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533965    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.534143 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.533990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-web-config\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.534143 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.534038    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.534626 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.534399    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.535264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.534917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.535417 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.535395    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.536882 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.536859    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-out\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.540963 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.540940    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.541305 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.541265    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.541405 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.541310    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.542003 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.541902    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.542116 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.542099    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.542190 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.542137    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.542392 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.542369    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.543849 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.543829    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx7qg\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-kube-api-access-hx7qg\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.543930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.543911    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-web-config\") pod \"alertmanager-main-0\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.688904 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.688868    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:19:42.939049 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.938997    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:42.983159 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:42.983125    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:19:43.012644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:43.012611    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:43.232514 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:19:43.232444    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56c8e0e_0fe1_473c_ac41_5bf770c9d2ab.slice/crio-9c8dbf7027fc732e97000b75da5ceecae8b044362c5f30825516f2440b377d69 WatchSource:0}: Error finding container 9c8dbf7027fc732e97000b75da5ceecae8b044362c5f30825516f2440b377d69: Status 404 returned error can't find the container with id 9c8dbf7027fc732e97000b75da5ceecae8b044362c5f30825516f2440b377d69
Sep 08 13:19:43.316395 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:43.316361    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp" event={"ID":"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8","Type":"ContainerStarted","Data":"bb86bc5b095b03fa40584afb9256be0935c9db7f1e2cc5dc83bf87508435a3aa"}
Sep 08 13:19:43.318435 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:43.318408    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerStarted","Data":"9c8dbf7027fc732e97000b75da5ceecae8b044362c5f30825516f2440b377d69"}
Sep 08 13:19:44.228568 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:44.228527    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:44.286622 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:44.286544    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:44.325124 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:44.325090    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ce5a7b7-4478-451d-b4fd-fc88df9f91de" containerID="74ce6ee232037b4e2d9d79df3ed4eaf2c91ec19360126f4937222d1f0eaba6ff" exitCode=0
Sep 08 13:19:44.325577 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:44.325147    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bnm9h" event={"ID":"3ce5a7b7-4478-451d-b4fd-fc88df9f91de","Type":"ContainerDied","Data":"74ce6ee232037b4e2d9d79df3ed4eaf2c91ec19360126f4937222d1f0eaba6ff"}
Sep 08 13:19:44.502447 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:44.501711    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:44.699170 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:44.699078    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:45.334347 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:45.334312    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bnm9h" event={"ID":"3ce5a7b7-4478-451d-b4fd-fc88df9f91de","Type":"ContainerStarted","Data":"466e3d9d1d5b36d5738efe70e7ec2ca36f1284ab2a54253005fe60e193418f04"}
Sep 08 13:19:45.334757 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:45.334380    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bnm9h" event={"ID":"3ce5a7b7-4478-451d-b4fd-fc88df9f91de","Type":"ContainerStarted","Data":"5cd43d3443c1c8a806312f6bcc679a69a1f870eb2d021fba4cb59d1a49a58e5a"}
Sep 08 13:19:45.336803 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:45.336708    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerStarted","Data":"b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25"}
Sep 08 13:19:45.339395 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:45.339371    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp" event={"ID":"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8","Type":"ContainerStarted","Data":"c016303583437e3a9e2e86b1b5e61446dc9de72e2509211bc4888e229d5e8fca"}
Sep 08 13:19:45.339497 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:45.339416    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp" event={"ID":"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8","Type":"ContainerStarted","Data":"09cef9db5f7d351dd9d92332b59274a1d6738b88117eed3a3474f4e0011d3425"}
Sep 08 13:19:45.339497 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:45.339436    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp" event={"ID":"90b4ca0f-0a3d-449c-86cf-0df165ac2ee8","Type":"ContainerStarted","Data":"c0f52dff199162b94d5eac5edc215c5c1cbb8bf7115acb69e3b5c957d55c2f68"}
Sep 08 13:19:45.355200 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:45.355139    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bnm9h" podStartSLOduration=3.257644624 podStartE2EDuration="4.355106573s" totalImagesPullingTime="1.019405117s" totalInitContainerRuntime="78.056832ms" isStatefulPod=true podCreationTimestamp="2026-09-08 13:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:45.352910215 +0000 UTC m=+214.281813994" watchObservedRunningTime="2026-09-08 13:19:45.355106573 +0000 UTC m=+214.284010314"
Sep 08 13:19:45.372053 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:45.371989    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-mdxjp" podStartSLOduration=2.478188165 podStartE2EDuration="4.371971397s" totalImagesPullingTime="1.893783232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:45.370102045 +0000 UTC m=+214.299005807" watchObservedRunningTime="2026-09-08 13:19:45.371971397 +0000 UTC m=+214.300875138"
Sep 08 13:19:46.033115 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:46.033079    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:46.120471 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:46.120436    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:46.210566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:46.210529    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:46.345642 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:46.345612    2583 generic.go:356] "Generic (PLEG): container finished" podID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerID="b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25" exitCode=0
Sep 08 13:19:46.346154 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:46.345702    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerDied","Data":"b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25"}
Sep 08 13:19:47.551161 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:47.551123    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:48.179169 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.179134    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7789bf9865-vzcqq"]
Sep 08 13:19:48.179491 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:19:48.179467    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7789bf9865-vzcqq" podUID="12f11884-1f2b-49bf-99b4-4c7f5f299fbe"
Sep 08 13:19:48.278518 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.278452    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:48.361165 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.361135    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:19:48.368382 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.368354    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:19:48.377944 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.377879    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:48.451256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.451135    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:48.492025 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.491976    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-installation-pull-secrets\") pod \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") "
Sep 08 13:19:48.492207 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492061    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-certificates\") pod \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") "
Sep 08 13:19:48.492207 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492113    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-bound-sa-token\") pod \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") "
Sep 08 13:19:48.492207 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492157    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xh82c\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-kube-api-access-xh82c\") pod \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") "
Sep 08 13:19:48.492378 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492333    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-image-registry-private-configuration\") pod \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") "
Sep 08 13:19:48.492442 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492421    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-trusted-ca\") pod \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") "
Sep 08 13:19:48.492492 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492480    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-ca-trust-extracted\") pod \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\" (UID: \"12f11884-1f2b-49bf-99b4-4c7f5f299fbe\") "
Sep 08 13:19:48.492543 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492502    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "12f11884-1f2b-49bf-99b4-4c7f5f299fbe" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:48.493001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492900    2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-certificates\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:19:48.493001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492951    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "12f11884-1f2b-49bf-99b4-4c7f5f299fbe" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:19:48.493001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.492975    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "12f11884-1f2b-49bf-99b4-4c7f5f299fbe" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:19:48.495198 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.494962    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "12f11884-1f2b-49bf-99b4-4c7f5f299fbe" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:48.495198 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.495179    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-kube-api-access-xh82c" (OuterVolumeSpecName: "kube-api-access-xh82c") pod "12f11884-1f2b-49bf-99b4-4c7f5f299fbe" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe"). InnerVolumeSpecName "kube-api-access-xh82c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:48.495345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.495277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "12f11884-1f2b-49bf-99b4-4c7f5f299fbe" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:19:48.495345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.495319    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "12f11884-1f2b-49bf-99b4-4c7f5f299fbe" (UID: "12f11884-1f2b-49bf-99b4-4c7f5f299fbe"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:19:48.594260 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.594221    2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-image-registry-private-configuration\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:19:48.594260 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.594264    2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-trusted-ca\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:19:48.594751 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.594280    2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-ca-trust-extracted\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:19:48.594751 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.594292    2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-installation-pull-secrets\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:19:48.594751 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.594307    2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-bound-sa-token\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:19:48.594751 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:48.594319    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh82c\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-kube-api-access-xh82c\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:19:49.369893 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:49.369862    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerStarted","Data":"7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde"}
Sep 08 13:19:49.369994 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:49.369948    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerStarted","Data":"08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16"}
Sep 08 13:19:49.369994 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:49.369983    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerStarted","Data":"0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c"}
Sep 08 13:19:49.370124 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:49.370042    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerStarted","Data":"b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7"}
Sep 08 13:19:49.370124 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:49.369873    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7789bf9865-vzcqq"
Sep 08 13:19:49.408653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:49.408616    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7789bf9865-vzcqq"]
Sep 08 13:19:49.412528 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:49.412504    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7789bf9865-vzcqq"]
Sep 08 13:19:49.503348 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:49.503315    2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/12f11884-1f2b-49bf-99b4-4c7f5f299fbe-registry-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:19:49.595141 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:49.595109    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12f11884-1f2b-49bf-99b4-4c7f5f299fbe" path="/var/lib/kubelet/pods/12f11884-1f2b-49bf-99b4-4c7f5f299fbe/volumes"
Sep 08 13:19:50.377697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:50.377663    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerStarted","Data":"127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647"}
Sep 08 13:19:52.311375 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:52.311341    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:52.373823 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:52.373784    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:53.263051 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:53.262997    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:53.361901 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:53.361856    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:53.673534 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:53.673494    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:53.740543 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:53.740516    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:54.897829 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:54.897794    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:55.417301 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:55.417251    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:55.517458 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:55.517423    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.409395 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:57.409357    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-nxwsj" event={"ID":"ec637e70-0655-4e0f-9da2-a4025cfd7e0f","Type":"ContainerStarted","Data":"af2aae03a59042d0018330d4f7e2334f1a1aa891957a4f3e7431c6e260fb470d"}
Sep 08 13:19:57.409858 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:57.409839    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-nxwsj"
Sep 08 13:19:57.421695 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:57.421670    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-nxwsj"
Sep 08 13:19:57.427630 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:57.427540    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-nxwsj" podStartSLOduration=1.792432364 podStartE2EDuration="19.42752521s" totalImagesPullingTime="17.635092846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:57.427028722 +0000 UTC m=+226.355932494" watchObservedRunningTime="2026-09-08 13:19:57.42752521 +0000 UTC m=+226.356428990"
Sep 08 13:19:57.464811 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:57.464771    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:57.558721 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:57.558692    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:58.155133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:58.155091    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:58.223427 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:58.223391    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:59.146958 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:59.146919    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:19:59.423473 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:59.423378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerStarted","Data":"88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc"}
Sep 08 13:19:59.454255 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:19:59.454203    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.412524754 podStartE2EDuration="17.454183128s" totalImagesPullingTime="14.022348405s" totalInitContainerRuntime="19.309969ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:19:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:19:59.453475338 +0000 UTC m=+228.382379096" watchObservedRunningTime="2026-09-08 13:19:59.454183128 +0000 UTC m=+228.383086871"
Sep 08 13:20:00.387697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:00.387645    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:02.181589 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:02.181528    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:02.302587 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:02.302549    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:04.493400 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:04.493361    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:04.554906 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:04.554867    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:08.661672 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:08.661641    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:08.727827 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:08.727764    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:09.871728 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:09.871691    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:09.934962 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:09.934909    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:14.148635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:14.148600    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:14.242082 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:14.242048    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:16.616784 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:16.616750    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:16.718288 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:16.718257    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:23.501557 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:23.501508    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:20:23.504121 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:23.504096    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29de17e1-0cb6-41fb-bd61-0a1fc39b19b3-metrics-certs\") pod \"network-metrics-daemon-82vzh\" (UID: \"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3\") " pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:20:23.793760 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:23.793681    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mxg9n\""
Sep 08 13:20:23.801757 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:23.801733    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-82vzh"
Sep 08 13:20:24.105021 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:24.104987    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-82vzh"]
Sep 08 13:20:24.108271 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:20:24.108245    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29de17e1_0cb6_41fb_bd61_0a1fc39b19b3.slice/crio-657463f48893e8bf8c1e57f4f941eeb26f58095b904ce2d0cf480bfbe3c42fad WatchSource:0}: Error finding container 657463f48893e8bf8c1e57f4f941eeb26f58095b904ce2d0cf480bfbe3c42fad: Status 404 returned error can't find the container with id 657463f48893e8bf8c1e57f4f941eeb26f58095b904ce2d0cf480bfbe3c42fad
Sep 08 13:20:24.519824 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:24.519732    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-82vzh" event={"ID":"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3","Type":"ContainerStarted","Data":"657463f48893e8bf8c1e57f4f941eeb26f58095b904ce2d0cf480bfbe3c42fad"}
Sep 08 13:20:26.528046 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:26.527992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-82vzh" event={"ID":"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3","Type":"ContainerStarted","Data":"5b87ba19ade003dcd0d2d7930a8e80312894eeb486cac3ff7b386effdcd88ada"}
Sep 08 13:20:26.528438 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:26.528056    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-82vzh" event={"ID":"29de17e1-0cb6-41fb-bd61-0a1fc39b19b3","Type":"ContainerStarted","Data":"a3de56d8b8d16a8812e5da1e5df51ca4ee1bd3b7c8fed327bf82ab136c8ece9b"}
Sep 08 13:20:26.544803 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:26.544755    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-82vzh" podStartSLOduration=253.248128132 podStartE2EDuration="4m14.544738679s" totalImagesPullingTime="1.296610547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:20:26.544550367 +0000 UTC m=+255.473454128" watchObservedRunningTime="2026-09-08 13:20:26.544738679 +0000 UTC m=+255.473642421"
Sep 08 13:20:26.646018 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:26.645969    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:26.717529 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:26.717464    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:26.890842 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:26.890811    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:26.990818 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:26.990788    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:27.091674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:27.091627    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:27.533494 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:27.533464    2583 generic.go:356] "Generic (PLEG): container finished" podID="da748d4c-3623-4746-b899-479c9aaa81c1" containerID="ba24ec77905dc35e873b24d211a019cbdb3b8d5d3ad2798b469aed605807b325" exitCode=0
Sep 08 13:20:27.533835 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:27.533533    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr" event={"ID":"da748d4c-3623-4746-b899-479c9aaa81c1","Type":"ContainerDied","Data":"ba24ec77905dc35e873b24d211a019cbdb3b8d5d3ad2798b469aed605807b325"}
Sep 08 13:20:27.534040 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:27.533999    2583 scope.go:117] "RemoveContainer" containerID="ba24ec77905dc35e873b24d211a019cbdb3b8d5d3ad2798b469aed605807b325"
Sep 08 13:20:28.539498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:28.539463    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-6lrnr" event={"ID":"da748d4c-3623-4746-b899-479c9aaa81c1","Type":"ContainerStarted","Data":"6820a01562447c777dc09c2111f4a5e871f40a4fe8129ad0fb6aa69ec97e5bf3"}
Sep 08 13:20:28.608782 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:28.608756    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9frn_02703e20-c856-4652-9e02-6086e7b11179/dns-node-resolver/0.log"
Sep 08 13:20:29.130680 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:29.130650    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:29.190738 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:29.190705    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:35.567369 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:35.567288    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:38.810415 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:38.810382    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:38.869119 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:38.869087    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:53.092914 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:20:53.092869    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-k6zh9" podUID="8e6529bc-e29e-406f-ae0e-55c85d81465b"
Sep 08 13:20:53.634591 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:53.634559    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:20:56.989099 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:56.989061    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:20:56.991526 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:56.991504    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e6529bc-e29e-406f-ae0e-55c85d81465b-metrics-tls\") pod \"dns-default-k6zh9\" (UID: \"8e6529bc-e29e-406f-ae0e-55c85d81465b\") " pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:20:57.237452 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:57.237421    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kwlvq\""
Sep 08 13:20:57.246150 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:57.246088    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:20:57.522237 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:57.522148    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k6zh9"]
Sep 08 13:20:57.525241 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:20:57.525214    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e6529bc_e29e_406f_ae0e_55c85d81465b.slice/crio-2b9cfaf8c1ff524ec95a2d84061c8f4fcff73675adbf724e312e07a4fbb31dbc WatchSource:0}: Error finding container 2b9cfaf8c1ff524ec95a2d84061c8f4fcff73675adbf724e312e07a4fbb31dbc: Status 404 returned error can't find the container with id 2b9cfaf8c1ff524ec95a2d84061c8f4fcff73675adbf724e312e07a4fbb31dbc
Sep 08 13:20:57.650313 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:57.650270    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6zh9" event={"ID":"8e6529bc-e29e-406f-ae0e-55c85d81465b","Type":"ContainerStarted","Data":"2b9cfaf8c1ff524ec95a2d84061c8f4fcff73675adbf724e312e07a4fbb31dbc"}
Sep 08 13:20:58.658307 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:58.658275    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:58.728702 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:58.728671    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:20:59.660925 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:20:59.660899    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6zh9" event={"ID":"8e6529bc-e29e-406f-ae0e-55c85d81465b","Type":"ContainerStarted","Data":"28b6ed52e95539bda95250784dfab7f66c667f95967fbaf5f4eeee09ce6ec483"}
Sep 08 13:21:00.571245 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:00.571212    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:00.667374 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:00.667339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k6zh9" event={"ID":"8e6529bc-e29e-406f-ae0e-55c85d81465b","Type":"ContainerStarted","Data":"afa9989ff0d17960229d570208ceca6099e17e5489b1f7f8aa08705bcc639808"}
Sep 08 13:21:00.667882 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:00.667598    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:21:00.685729 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:00.685665    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k6zh9" podStartSLOduration=252.650692083 podStartE2EDuration="4m14.685644192s" totalImagesPullingTime="2.034952109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:00.684440745 +0000 UTC m=+289.613344506" watchObservedRunningTime="2026-09-08 13:21:00.685644192 +0000 UTC m=+289.614547935"
Sep 08 13:21:00.916786 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:00.916701    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:01.683553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:01.683521    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:01.683979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:01.683801    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="alertmanager" containerID="cri-o://b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7" gracePeriod=120
Sep 08 13:21:01.683979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:01.683881    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy-metric" containerID="cri-o://127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647" gracePeriod=120
Sep 08 13:21:01.683979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:01.683905    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="config-reloader" containerID="cri-o://0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c" gracePeriod=120
Sep 08 13:21:01.683979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:01.683933    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="prom-label-proxy" containerID="cri-o://88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc" gracePeriod=120
Sep 08 13:21:01.683979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:01.683957    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy" containerID="cri-o://7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde" gracePeriod=120
Sep 08 13:21:01.683979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:01.683881    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy-web" containerID="cri-o://08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16" gracePeriod=120
Sep 08 13:21:02.680668 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680634    2583 generic.go:356] "Generic (PLEG): container finished" podID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerID="88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc" exitCode=0
Sep 08 13:21:02.680668 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680659    2583 generic.go:356] "Generic (PLEG): container finished" podID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerID="127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647" exitCode=0
Sep 08 13:21:02.680668 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680667    2583 generic.go:356] "Generic (PLEG): container finished" podID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerID="7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde" exitCode=0
Sep 08 13:21:02.680668 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680673    2583 generic.go:356] "Generic (PLEG): container finished" podID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerID="0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c" exitCode=0
Sep 08 13:21:02.680668 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680680    2583 generic.go:356] "Generic (PLEG): container finished" podID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerID="b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7" exitCode=0
Sep 08 13:21:02.680961 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680701    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerDied","Data":"88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc"}
Sep 08 13:21:02.680961 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680764    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerDied","Data":"127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647"}
Sep 08 13:21:02.680961 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680791    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerDied","Data":"7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde"}
Sep 08 13:21:02.680961 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680816    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerDied","Data":"0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c"}
Sep 08 13:21:02.680961 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.680840    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerDied","Data":"b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7"}
Sep 08 13:21:02.933814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:02.933739    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.034618 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.034586    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-cluster-tls-config\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.034618 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.034624    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-tls-assets\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.034853 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.034648    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-main-tls\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.034853 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.034676    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx7qg\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-kube-api-access-hx7qg\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.034853 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.034698    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-volume\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.034853 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.034721    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.034853 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.034756    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.037305 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.037269    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-kube-api-access-hx7qg" (OuterVolumeSpecName: "kube-api-access-hx7qg") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "kube-api-access-hx7qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:21:03.037432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.037418    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-volume" (OuterVolumeSpecName: "config-volume") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:03.037778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.037743    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:03.037889 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.037780    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:03.038052 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.037995    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:03.038052 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.038042    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:21:03.041535 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.041468    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:03.135416 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135378    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-out\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.135596 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135424    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.135596 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135448    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-metrics-client-ca\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.135596 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135467    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-web-config\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.135596 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135484    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-trusted-ca-bundle\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.135596 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135499    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-main-db\") pod \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\" (UID: \"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab\") "
Sep 08 13:21:03.135867 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135705    2583 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-cluster-tls-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.135867 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135722    2583 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-tls-assets\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.135867 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135735    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-main-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.135867 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135748    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx7qg\" (UniqueName: \"kubernetes.io/projected/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-kube-api-access-hx7qg\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.135867 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135761    2583 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-volume\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.135867 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135776    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.135867 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135792    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.136196 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.135911    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:03.136196 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.136061    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:21:03.136196 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.136079    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:21:03.137758 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.137738    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-out" (OuterVolumeSpecName: "config-out") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:21:03.137853 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.137804    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:03.147401 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.147372    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-web-config" (OuterVolumeSpecName: "web-config") pod "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" (UID: "d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:21:03.240256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.237229    2583 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-config-out\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.240256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.237277    2583 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.240256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.237292    2583 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-metrics-client-ca\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.240256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.237306    2583 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-web-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.240256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.237321    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.240256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.237339    2583 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab-alertmanager-main-db\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:21:03.688255 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.688221    2583 generic.go:356] "Generic (PLEG): container finished" podID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerID="08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16" exitCode=0
Sep 08 13:21:03.688449 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.688296    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerDied","Data":"08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16"}
Sep 08 13:21:03.688449 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.688338    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.688449 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.688373    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab","Type":"ContainerDied","Data":"9c8dbf7027fc732e97000b75da5ceecae8b044362c5f30825516f2440b377d69"}
Sep 08 13:21:03.688449 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.688406    2583 scope.go:117] "RemoveContainer" containerID="88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc"
Sep 08 13:21:03.696815 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.696796    2583 scope.go:117] "RemoveContainer" containerID="127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647"
Sep 08 13:21:03.704593 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.704570    2583 scope.go:117] "RemoveContainer" containerID="7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde"
Sep 08 13:21:03.712256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.712224    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:03.713108 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.713088    2583 scope.go:117] "RemoveContainer" containerID="08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16"
Sep 08 13:21:03.716777 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.716749    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:03.721435 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.721417    2583 scope.go:117] "RemoveContainer" containerID="0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c"
Sep 08 13:21:03.729642 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.729618    2583 scope.go:117] "RemoveContainer" containerID="b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7"
Sep 08 13:21:03.738079 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.738051    2583 scope.go:117] "RemoveContainer" containerID="b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25"
Sep 08 13:21:03.746481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.746454    2583 scope.go:117] "RemoveContainer" containerID="88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc"
Sep 08 13:21:03.746890 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:21:03.746866    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc\": container with ID starting with 88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc not found: ID does not exist" containerID="88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc"
Sep 08 13:21:03.746984 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.746899    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc"} err="failed to get container status \"88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc\": rpc error: code = NotFound desc = could not find container \"88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc\": container with ID starting with 88fc112b40d30c14cbcf8fc6b97c4273bae92105009e7960a508ddef39cb0ebc not found: ID does not exist"
Sep 08 13:21:03.746984 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.746933    2583 scope.go:117] "RemoveContainer" containerID="127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647"
Sep 08 13:21:03.747265 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:21:03.747243    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647\": container with ID starting with 127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647 not found: ID does not exist" containerID="127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647"
Sep 08 13:21:03.747309 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747275    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647"} err="failed to get container status \"127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647\": rpc error: code = NotFound desc = could not find container \"127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647\": container with ID starting with 127b89bfaacf98f2c4292aed2f47c11b7e4e46924d9b49b6657f55181d767647 not found: ID does not exist"
Sep 08 13:21:03.747309 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747299    2583 scope.go:117] "RemoveContainer" containerID="7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde"
Sep 08 13:21:03.747749 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747536    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:03.747749 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:21:03.747563    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde\": container with ID starting with 7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde not found: ID does not exist" containerID="7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde"
Sep 08 13:21:03.747749 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747612    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde"} err="failed to get container status \"7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde\": rpc error: code = NotFound desc = could not find container \"7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde\": container with ID starting with 7ae889a7a0177cf8189f68d33f2eccb2615f61a18af0d1f6fedb1b2f907f7dde not found: ID does not exist"
Sep 08 13:21:03.747749 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747631    2583 scope.go:117] "RemoveContainer" containerID="08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747890    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy-metric"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747903    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy-metric"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747912    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="config-reloader"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747920    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="config-reloader"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747935    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy-web"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747943    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy-web"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747955    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747963    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747971    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="init-config-reloader"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747976    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="init-config-reloader"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:21:03.747979    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16\": container with ID starting with 08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16 not found: ID does not exist" containerID="08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16"
Sep 08 13:21:03.748030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.747988    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="prom-label-proxy"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748042    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="prom-label-proxy"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748049    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="alertmanager"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748053    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="alertmanager"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748106    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy-metric"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748115    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748120    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="alertmanager"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748127    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="config-reloader"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748132    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="prom-label-proxy"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748136    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" containerName="kube-rbac-proxy-web"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748002    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16"} err="failed to get container status \"08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16\": rpc error: code = NotFound desc = could not find container \"08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16\": container with ID starting with 08f017db6edac22593e9933eb1949d030ce9f8b07fd330db4744e0c2ebdacb16 not found: ID does not exist"
Sep 08 13:21:03.748581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748228    2583 scope.go:117] "RemoveContainer" containerID="0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c"
Sep 08 13:21:03.749244 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:21:03.748752    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c\": container with ID starting with 0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c not found: ID does not exist" containerID="0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c"
Sep 08 13:21:03.749244 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748772    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c"} err="failed to get container status \"0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c\": rpc error: code = NotFound desc = could not find container \"0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c\": container with ID starting with 0200c6c38d71f60b174f53cee0eedab2c7adbc9ebdad03755214cc9696c7f48c not found: ID does not exist"
Sep 08 13:21:03.749244 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.748787    2583 scope.go:117] "RemoveContainer" containerID="b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7"
Sep 08 13:21:03.749244 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:21:03.749101    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7\": container with ID starting with b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7 not found: ID does not exist" containerID="b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7"
Sep 08 13:21:03.749244 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.749140    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7"} err="failed to get container status \"b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7\": rpc error: code = NotFound desc = could not find container \"b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7\": container with ID starting with b5eb88f40640d56250fa9823d2549b6135bbf89d4010315d77d0b9de68940ea7 not found: ID does not exist"
Sep 08 13:21:03.749244 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.749161    2583 scope.go:117] "RemoveContainer" containerID="b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25"
Sep 08 13:21:03.749608 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:21:03.749469    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25\": container with ID starting with b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25 not found: ID does not exist" containerID="b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25"
Sep 08 13:21:03.749608 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.749498    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25"} err="failed to get container status \"b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25\": rpc error: code = NotFound desc = could not find container \"b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25\": container with ID starting with b7928ae9addccc81958551f1f9123a54bc498322efcaee1e0d3d3567a56fee25 not found: ID does not exist"
Sep 08 13:21:03.752608 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.752588    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.754684 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.754660    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 08 13:21:03.756292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.756274    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 08 13:21:03.756587 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.756570    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 08 13:21:03.756854 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.756836    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zhnnr\""
Sep 08 13:21:03.756968 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.756950    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 08 13:21:03.757123 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.757098    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 08 13:21:03.757239 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.757102    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 08 13:21:03.757239 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.757213    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 08 13:21:03.757345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.757267    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 08 13:21:03.762308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.762280    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 08 13:21:03.767907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.767874    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:03.842062 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.841995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b45e756-c32f-46e3-8cee-11ef1c8e583b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842262 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842072    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b45e756-c32f-46e3-8cee-11ef1c8e583b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842262 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842108    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842262 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842131    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b45e756-c32f-46e3-8cee-11ef1c8e583b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842262 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842152    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pcdb\" (UniqueName: \"kubernetes.io/projected/9b45e756-c32f-46e3-8cee-11ef1c8e583b-kube-api-access-6pcdb\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842262 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842178    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842262 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842214    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842262 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842255    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-config-volume\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842522 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842272    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-web-config\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842522 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842304    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b45e756-c32f-46e3-8cee-11ef1c8e583b-config-out\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842522 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842327    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842522 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842353    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.842522 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.842388    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b45e756-c32f-46e3-8cee-11ef1c8e583b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.943605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943521    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.943605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-config-volume\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.943605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943583    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-web-config\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944229 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943624    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b45e756-c32f-46e3-8cee-11ef1c8e583b-config-out\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944229 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943731    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944229 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943775    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944229 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943838    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b45e756-c32f-46e3-8cee-11ef1c8e583b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944229 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b45e756-c32f-46e3-8cee-11ef1c8e583b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944229 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943905    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b45e756-c32f-46e3-8cee-11ef1c8e583b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944704 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.944260    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9b45e756-c32f-46e3-8cee-11ef1c8e583b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944704 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.943932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944704 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.944305    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b45e756-c32f-46e3-8cee-11ef1c8e583b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944704 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.944332    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcdb\" (UniqueName: \"kubernetes.io/projected/9b45e756-c32f-46e3-8cee-11ef1c8e583b-kube-api-access-6pcdb\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.944704 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.944499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.946446 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.945112    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b45e756-c32f-46e3-8cee-11ef1c8e583b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.946446 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.946085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9b45e756-c32f-46e3-8cee-11ef1c8e583b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.946878 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.946854    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.946994 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.946914    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.946994 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.946979    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9b45e756-c32f-46e3-8cee-11ef1c8e583b-config-out\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.947146 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.947117    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-web-config\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.947593 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.947574    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.947908 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.947888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9b45e756-c32f-46e3-8cee-11ef1c8e583b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.948154 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.948135    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-config-volume\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.948637 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.948620    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.949182 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.949162    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9b45e756-c32f-46e3-8cee-11ef1c8e583b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:03.956656 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:03.956626    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pcdb\" (UniqueName: \"kubernetes.io/projected/9b45e756-c32f-46e3-8cee-11ef1c8e583b-kube-api-access-6pcdb\") pod \"alertmanager-main-0\" (UID: \"9b45e756-c32f-46e3-8cee-11ef1c8e583b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:04.062970 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:04.062918    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 08 13:21:04.346287 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:04.346265    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 08 13:21:04.348657 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:21:04.348630    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b45e756_c32f_46e3_8cee_11ef1c8e583b.slice/crio-437244df745a06ead651a0183a5cf426d421ec505cb80321c2acea88e0014fd0 WatchSource:0}: Error finding container 437244df745a06ead651a0183a5cf426d421ec505cb80321c2acea88e0014fd0: Status 404 returned error can't find the container with id 437244df745a06ead651a0183a5cf426d421ec505cb80321c2acea88e0014fd0
Sep 08 13:21:04.693986 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:04.693901    2583 generic.go:356] "Generic (PLEG): container finished" podID="9b45e756-c32f-46e3-8cee-11ef1c8e583b" containerID="836bf5cb8d7f5c0c331c6a3bae6911abe9e679674df2aff147335ba2e88dfa95" exitCode=0
Sep 08 13:21:04.693986 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:04.693941    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b45e756-c32f-46e3-8cee-11ef1c8e583b","Type":"ContainerDied","Data":"836bf5cb8d7f5c0c331c6a3bae6911abe9e679674df2aff147335ba2e88dfa95"}
Sep 08 13:21:04.694174 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:04.693991    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b45e756-c32f-46e3-8cee-11ef1c8e583b","Type":"ContainerStarted","Data":"437244df745a06ead651a0183a5cf426d421ec505cb80321c2acea88e0014fd0"}
Sep 08 13:21:05.594150 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.594115    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab" path="/var/lib/kubelet/pods/d56c8e0e-0fe1-473c-ac41-5bf770c9d2ab/volumes"
Sep 08 13:21:05.692617 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.692584    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"]
Sep 08 13:21:05.695163 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.695142    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.697445 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.697419    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 08 13:21:05.697602 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.697581    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 08 13:21:05.697602 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.697591    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7ndcn\""
Sep 08 13:21:05.697833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.697417    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 08 13:21:05.698584 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.698563    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 08 13:21:05.698679 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.698652    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 08 13:21:05.703753 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.703728    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 08 13:21:05.704426 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.704403    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b45e756-c32f-46e3-8cee-11ef1c8e583b","Type":"ContainerStarted","Data":"540eaebe6e9978591c63fe35dcfb3a70be614c7ff50df3533d1a0075f3818386"}
Sep 08 13:21:05.704511 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.704482    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b45e756-c32f-46e3-8cee-11ef1c8e583b","Type":"ContainerStarted","Data":"49ca6508878184f6e22f4295f64de1e63d0643221dfa88e025aacea844730e30"}
Sep 08 13:21:05.704567 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.704526    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b45e756-c32f-46e3-8cee-11ef1c8e583b","Type":"ContainerStarted","Data":"c1b55d07f1628921abeecd553f8c9d783f74619136ecd1450131cad595c0ac88"}
Sep 08 13:21:05.704628 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.704568    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b45e756-c32f-46e3-8cee-11ef1c8e583b","Type":"ContainerStarted","Data":"da2e0288cf8cc52f838ae2c0259db09de6db42b428a71e46816c60b376a7e5af"}
Sep 08 13:21:05.704628 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.704610    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b45e756-c32f-46e3-8cee-11ef1c8e583b","Type":"ContainerStarted","Data":"e354410b4ee3b1fe830021c35ae33c377468f7f1e0f50987d4884ff52da31f13"}
Sep 08 13:21:05.704727 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.704650    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9b45e756-c32f-46e3-8cee-11ef1c8e583b","Type":"ContainerStarted","Data":"c53c9c5a3dba0005ade6cdd554d8d3885f4e09e1c8e1da2190b071b2f0c82cb1"}
Sep 08 13:21:05.709952 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.709918    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"]
Sep 08 13:21:05.761809 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.761770    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsrsw\" (UniqueName: \"kubernetes.io/projected/63c6e89b-db17-4ab1-b8bf-815a19936cff-kube-api-access-qsrsw\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.761809 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.761813    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-secret-telemeter-client\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.762053 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.761851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c6e89b-db17-4ab1-b8bf-815a19936cff-serving-certs-ca-bundle\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.762053 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.761998    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-telemeter-client-tls\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.762271 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.762253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.762351 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.762336    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c6e89b-db17-4ab1-b8bf-815a19936cff-metrics-client-ca\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.762470 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.762446    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-federate-client-tls\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.762630 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.762613    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c6e89b-db17-4ab1-b8bf-815a19936cff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.766198 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.766150    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.750219724 podStartE2EDuration="2.766134534s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.91481ms" isStatefulPod=false podCreationTimestamp="2026-09-08 13:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:05.764253299 +0000 UTC m=+294.693157042" watchObservedRunningTime="2026-09-08 13:21:05.766134534 +0000 UTC m=+294.695038277"
Sep 08 13:21:05.863398 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.863286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c6e89b-db17-4ab1-b8bf-815a19936cff-metrics-client-ca\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.863398 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.863335    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-federate-client-tls\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.863398 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.863371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c6e89b-db17-4ab1-b8bf-815a19936cff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.863398 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.863402    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsrsw\" (UniqueName: \"kubernetes.io/projected/63c6e89b-db17-4ab1-b8bf-815a19936cff-kube-api-access-qsrsw\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.863738 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.863434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-secret-telemeter-client\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.863738 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.863585    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c6e89b-db17-4ab1-b8bf-815a19936cff-serving-certs-ca-bundle\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.863738 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.863637    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-telemeter-client-tls\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.863738 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.863705    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.864878 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.864183    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c6e89b-db17-4ab1-b8bf-815a19936cff-metrics-client-ca\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.864878 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.864334    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c6e89b-db17-4ab1-b8bf-815a19936cff-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.864878 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.864619    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c6e89b-db17-4ab1-b8bf-815a19936cff-serving-certs-ca-bundle\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.867083 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.867040    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-telemeter-client-tls\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.867530 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.867500    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.867937 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.867919    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-secret-telemeter-client\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.867985 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.867949    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/63c6e89b-db17-4ab1-b8bf-815a19936cff-federate-client-tls\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:05.874577 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:05.874554    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsrsw\" (UniqueName: \"kubernetes.io/projected/63c6e89b-db17-4ab1-b8bf-815a19936cff-kube-api-access-qsrsw\") pod \"telemeter-client-79bf894ccb-sx6wb\" (UID: \"63c6e89b-db17-4ab1-b8bf-815a19936cff\") " pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:06.008000 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:06.007966    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"
Sep 08 13:21:06.301303 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:06.301214    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79bf894ccb-sx6wb"]
Sep 08 13:21:06.305197 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:21:06.305164    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c6e89b_db17_4ab1_b8bf_815a19936cff.slice/crio-af974cde95ee158cf17161e9e6f8665c0ec475adeb7dc2fed7e3fb6d6dc86809 WatchSource:0}: Error finding container af974cde95ee158cf17161e9e6f8665c0ec475adeb7dc2fed7e3fb6d6dc86809: Status 404 returned error can't find the container with id af974cde95ee158cf17161e9e6f8665c0ec475adeb7dc2fed7e3fb6d6dc86809
Sep 08 13:21:06.709663 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:06.709625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb" event={"ID":"63c6e89b-db17-4ab1-b8bf-815a19936cff","Type":"ContainerStarted","Data":"af974cde95ee158cf17161e9e6f8665c0ec475adeb7dc2fed7e3fb6d6dc86809"}
Sep 08 13:21:08.726139 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:08.726105    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb" event={"ID":"63c6e89b-db17-4ab1-b8bf-815a19936cff","Type":"ContainerStarted","Data":"003367672fe847273ced799003ac8e3827e95240a2d8ee91a1ad5a9b1414886f"}
Sep 08 13:21:08.726601 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:08.726182    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb" event={"ID":"63c6e89b-db17-4ab1-b8bf-815a19936cff","Type":"ContainerStarted","Data":"9804234b049d0da91bff50365e1654ee3d45e5cb0353b57763e3940256f53372"}
Sep 08 13:21:08.726601 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:08.726204    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb" event={"ID":"63c6e89b-db17-4ab1-b8bf-815a19936cff","Type":"ContainerStarted","Data":"d377eff9f50441c8e217583d91833d06d7c2ff3f7482f401a311125a9fa0607e"}
Sep 08 13:21:08.751551 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:08.751484    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79bf894ccb-sx6wb" podStartSLOduration=1.7840978029999999 podStartE2EDuration="3.751459433s" totalImagesPullingTime="1.96736163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:21:08.750081759 +0000 UTC m=+297.678985519" watchObservedRunningTime="2026-09-08 13:21:08.751459433 +0000 UTC m=+297.680363178"
Sep 08 13:21:08.922570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:08.922537    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:09.000106 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:09.000004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:10.674521 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:10.674488    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k6zh9"
Sep 08 13:21:11.503957 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:11.503928    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:21:11.504225 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:11.504204    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:21:11.516339 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:11.516320    2583 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 13:21:19.930279 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:19.930245    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:20.057207 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:20.057172    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:28.867128 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:28.867094    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:28.936177 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:28.936136    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:58.479650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:58.479615    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:21:58.576541 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:21:58.576506    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:25.877098 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:25.877004    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:25.954023 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:25.953991    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:28.984879 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:28.984845    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:30.069312 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:30.069276    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:30.139082 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:30.139051    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:37.966335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:37.966297    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:38.023526 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:38.023493    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:41.111416 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:41.111382    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:41.211126 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:41.211093    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:41.289861 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:41.289833    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:44.143281 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:44.143254    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:45.680478 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:45.680439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:22:45.761030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:22:45.760984    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:23:15.316877 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:23:15.316839    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:26.085520 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:26.085485    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:29.624431 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:29.624392    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:48.664280 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:48.664244    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:48.774834 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:48.774807    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:50.317984 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.317943    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"]
Sep 08 13:24:50.321297 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.321277    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:24:50.323824 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.323804    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 08 13:24:50.323944 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.323839    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 08 13:24:50.324179 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.324163    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-hzm6w\""
Sep 08 13:24:50.324357 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.324344    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 08 13:24:50.337956 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.337925    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"]
Sep 08 13:24:50.348351 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.348322    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/34524880-6a7a-4ae2-9352-07db665ae491-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ntm8v\" (UID: \"34524880-6a7a-4ae2-9352-07db665ae491\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:24:50.348496 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.348451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jskh8\" (UniqueName: \"kubernetes.io/projected/34524880-6a7a-4ae2-9352-07db665ae491-kube-api-access-jskh8\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ntm8v\" (UID: \"34524880-6a7a-4ae2-9352-07db665ae491\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:24:50.449227 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.449189    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jskh8\" (UniqueName: \"kubernetes.io/projected/34524880-6a7a-4ae2-9352-07db665ae491-kube-api-access-jskh8\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ntm8v\" (UID: \"34524880-6a7a-4ae2-9352-07db665ae491\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:24:50.449409 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.449245    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/34524880-6a7a-4ae2-9352-07db665ae491-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ntm8v\" (UID: \"34524880-6a7a-4ae2-9352-07db665ae491\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:24:50.451915 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.451875    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/34524880-6a7a-4ae2-9352-07db665ae491-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ntm8v\" (UID: \"34524880-6a7a-4ae2-9352-07db665ae491\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:24:50.459509 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.459475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jskh8\" (UniqueName: \"kubernetes.io/projected/34524880-6a7a-4ae2-9352-07db665ae491-kube-api-access-jskh8\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-ntm8v\" (UID: \"34524880-6a7a-4ae2-9352-07db665ae491\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:24:50.631612 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.631517    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:24:50.912796 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.912723    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"]
Sep 08 13:24:50.916508 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:24:50.916481    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34524880_6a7a_4ae2_9352_07db665ae491.slice/crio-ab02d20ca307e82be8c7fda435b3f5a9c3732bb9f78758282b2ff17f648da45d WatchSource:0}: Error finding container ab02d20ca307e82be8c7fda435b3f5a9c3732bb9f78758282b2ff17f648da45d: Status 404 returned error can't find the container with id ab02d20ca307e82be8c7fda435b3f5a9c3732bb9f78758282b2ff17f648da45d
Sep 08 13:24:50.918390 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:50.918371    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:24:51.556148 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:51.556095    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v" event={"ID":"34524880-6a7a-4ae2-9352-07db665ae491","Type":"ContainerStarted","Data":"ab02d20ca307e82be8c7fda435b3f5a9c3732bb9f78758282b2ff17f648da45d"}
Sep 08 13:24:54.251717 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:54.251616    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:54.349411 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:54.349375    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:54.449205 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:54.449169    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:54.567964 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:54.567876    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:54.574866 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:54.574835    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v" event={"ID":"34524880-6a7a-4ae2-9352-07db665ae491","Type":"ContainerStarted","Data":"c1b4d01b0764fb69dbb7d7ba3c9ce12b4e45985ebd259b17800b8cf54f7007a0"}
Sep 08 13:24:54.575026 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:54.574937    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:24:54.607247 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:54.607190    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v" podStartSLOduration=1.5690530919999999 podStartE2EDuration="4.607169235s" totalImagesPullingTime="3.038116143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:54.605826804 +0000 UTC m=+523.534730573" watchObservedRunningTime="2026-09-08 13:24:54.607169235 +0000 UTC m=+523.536072979"
Sep 08 13:24:54.788726 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:54.788698    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:55.021679 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.021584    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"]
Sep 08 13:24:55.025365 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.025345    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.027412 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.027387    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 08 13:24:55.027521 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.027504    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vk7qr\""
Sep 08 13:24:55.027744 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.027503    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 08 13:24:55.036454 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.036390    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:55.036598 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.036470    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"]
Sep 08 13:24:55.090072 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.090003    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgdg\" (UniqueName: \"kubernetes.io/projected/e8d9a402-73d8-4050-a347-98a3b9a16db6-kube-api-access-4sgdg\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.090263 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.090128    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e8d9a402-73d8-4050-a347-98a3b9a16db6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.090263 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.090160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e8d9a402-73d8-4050-a347-98a3b9a16db6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.098025 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.097981    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:55.191003 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.190968    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e8d9a402-73d8-4050-a347-98a3b9a16db6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.191196 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.191005    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e8d9a402-73d8-4050-a347-98a3b9a16db6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.191196 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.191101    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgdg\" (UniqueName: \"kubernetes.io/projected/e8d9a402-73d8-4050-a347-98a3b9a16db6-kube-api-access-4sgdg\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.191196 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:24:55.191124    2583 secret.go:281] references non-existent secret key: tls.crt
Sep 08 13:24:55.191196 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:24:55.191142    2583 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 08 13:24:55.191196 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:24:55.191162    2583 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt: references non-existent secret key: tls.crt
Sep 08 13:24:55.191398 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:24:55.191214    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d9a402-73d8-4050-a347-98a3b9a16db6-certificates podName:e8d9a402-73d8-4050-a347-98a3b9a16db6 nodeName:}" failed. No retries permitted until 2026-09-08 13:24:55.691199222 +0000 UTC m=+524.620102942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/e8d9a402-73d8-4050-a347-98a3b9a16db6-certificates") pod "keda-metrics-apiserver-77857b7b98-5s7gt" (UID: "e8d9a402-73d8-4050-a347-98a3b9a16db6") : references non-existent secret key: tls.crt
Sep 08 13:24:55.191398 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.191391    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e8d9a402-73d8-4050-a347-98a3b9a16db6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.199127 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.199100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgdg\" (UniqueName: \"kubernetes.io/projected/e8d9a402-73d8-4050-a347-98a3b9a16db6-kube-api-access-4sgdg\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.595588 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.595556    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-7rlzr"]
Sep 08 13:24:55.599190 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.599164    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:24:55.600955 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.600934    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 08 13:24:55.610181 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.610153    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-7rlzr"]
Sep 08 13:24:55.695383 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.695340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9a774721-06cf-453c-8f0b-b19d68fdc22e-certificates\") pod \"keda-admission-6c4d864b86-7rlzr\" (UID: \"9a774721-06cf-453c-8f0b-b19d68fdc22e\") " pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:24:55.695559 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.695430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e8d9a402-73d8-4050-a347-98a3b9a16db6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.695559 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.695459    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbxcr\" (UniqueName: \"kubernetes.io/projected/9a774721-06cf-453c-8f0b-b19d68fdc22e-kube-api-access-bbxcr\") pod \"keda-admission-6c4d864b86-7rlzr\" (UID: \"9a774721-06cf-453c-8f0b-b19d68fdc22e\") " pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:24:55.698924 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.698872    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e8d9a402-73d8-4050-a347-98a3b9a16db6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-5s7gt\" (UID: \"e8d9a402-73d8-4050-a347-98a3b9a16db6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:55.796801 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.796755    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9a774721-06cf-453c-8f0b-b19d68fdc22e-certificates\") pod \"keda-admission-6c4d864b86-7rlzr\" (UID: \"9a774721-06cf-453c-8f0b-b19d68fdc22e\") " pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:24:55.797006 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.796894    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbxcr\" (UniqueName: \"kubernetes.io/projected/9a774721-06cf-453c-8f0b-b19d68fdc22e-kube-api-access-bbxcr\") pod \"keda-admission-6c4d864b86-7rlzr\" (UID: \"9a774721-06cf-453c-8f0b-b19d68fdc22e\") " pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:24:55.799571 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.799537    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/9a774721-06cf-453c-8f0b-b19d68fdc22e-certificates\") pod \"keda-admission-6c4d864b86-7rlzr\" (UID: \"9a774721-06cf-453c-8f0b-b19d68fdc22e\") " pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:24:55.807727 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.807695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbxcr\" (UniqueName: \"kubernetes.io/projected/9a774721-06cf-453c-8f0b-b19d68fdc22e-kube-api-access-bbxcr\") pod \"keda-admission-6c4d864b86-7rlzr\" (UID: \"9a774721-06cf-453c-8f0b-b19d68fdc22e\") " pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:24:55.910228 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.910129    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:24:55.937190 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:55.937153    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:56.216253 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:56.213872    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-7rlzr"]
Sep 08 13:24:56.251747 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:56.251723    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"]
Sep 08 13:24:56.254495 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:24:56.254465    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d9a402_73d8_4050_a347_98a3b9a16db6.slice/crio-ca6dc5017d64f2d8b2fc692fed65be3f361bcb0f074e2016183830a79ad374b6 WatchSource:0}: Error finding container ca6dc5017d64f2d8b2fc692fed65be3f361bcb0f074e2016183830a79ad374b6: Status 404 returned error can't find the container with id ca6dc5017d64f2d8b2fc692fed65be3f361bcb0f074e2016183830a79ad374b6
Sep 08 13:24:56.429675 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:56.429641    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:56.531062 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:56.530949    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:56.586631 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:56.586594    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-7rlzr" event={"ID":"9a774721-06cf-453c-8f0b-b19d68fdc22e","Type":"ContainerStarted","Data":"271034b6507667ced111d1d4be7f420d8b146539c6c6c4bf43946392319b40f9"}
Sep 08 13:24:56.588087 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:56.588058    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt" event={"ID":"e8d9a402-73d8-4050-a347-98a3b9a16db6","Type":"ContainerStarted","Data":"ca6dc5017d64f2d8b2fc692fed65be3f361bcb0f074e2016183830a79ad374b6"}
Sep 08 13:24:57.108666 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:57.108627    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:57.207724 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:57.207690    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:24:57.596715 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:57.596686    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:24:57.596876 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:57.596751    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-7rlzr" event={"ID":"9a774721-06cf-453c-8f0b-b19d68fdc22e","Type":"ContainerStarted","Data":"faad54f24c4da49345d3f19289651c2e52d993c2a518d9bd64904e6c8a308262"}
Sep 08 13:24:57.617577 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:57.617508    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-7rlzr" podStartSLOduration=1.420548804 podStartE2EDuration="2.617487268s" totalImagesPullingTime="1.196938464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:57.61567819 +0000 UTC m=+526.544581944" watchObservedRunningTime="2026-09-08 13:24:57.617487268 +0000 UTC m=+526.546391012"
Sep 08 13:24:59.606059 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:59.606002    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt" event={"ID":"e8d9a402-73d8-4050-a347-98a3b9a16db6","Type":"ContainerStarted","Data":"9fcff218eb23f8ac3c6c157f2ee1b7d816a85f823e98eacf3f56e631fef93199"}
Sep 08 13:24:59.606522 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:59.606145    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:24:59.625500 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:24:59.625439    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt" podStartSLOduration=2.242517085 podStartE2EDuration="4.625420412s" totalImagesPullingTime="2.382903327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:24:59.624294118 +0000 UTC m=+528.553197890" watchObservedRunningTime="2026-09-08 13:24:59.625420412 +0000 UTC m=+528.554324450"
Sep 08 13:25:03.269112 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:03.269077    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:03.339071 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:03.339041    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:04.726916 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:04.726879    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:04.826147 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:04.826116    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:06.025985 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:06.025942    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:10.615149 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:10.615123    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-5s7gt"
Sep 08 13:25:15.582653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:15.582624    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-ntm8v"
Sep 08 13:25:18.601619 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:18.601592    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-7rlzr"
Sep 08 13:25:47.311421 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:47.311362    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:47.406256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:47.406207    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:25:51.414039 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:25:51.413987    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:00.470362 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:00.470311    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:00.580180 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:00.580141    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:11.531125 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:11.531096    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:26:11.532686 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:11.532665    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:26:13.130875 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.130842    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:13.189751 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.189718    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-jnrsc"]
Sep 08 13:26:13.192175 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.192139    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:13.194128 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.194104    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 08 13:26:13.194333 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.194309    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 08 13:26:13.194654 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.194642    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-j7p2q\""
Sep 08 13:26:13.194716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.194660    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 08 13:26:13.205330 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.205293    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-jnrsc"]
Sep 08 13:26:13.262125 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.262094    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgc8h\" (UniqueName: \"kubernetes.io/projected/fec8002a-2dce-4c5a-b601-5fdd2ac00461-kube-api-access-pgc8h\") pod \"kserve-controller-manager-575d6764f4-jnrsc\" (UID: \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\") " pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:13.262319 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.262180    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fec8002a-2dce-4c5a-b601-5fdd2ac00461-cert\") pod \"kserve-controller-manager-575d6764f4-jnrsc\" (UID: \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\") " pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:13.363003 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.362959    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgc8h\" (UniqueName: \"kubernetes.io/projected/fec8002a-2dce-4c5a-b601-5fdd2ac00461-kube-api-access-pgc8h\") pod \"kserve-controller-manager-575d6764f4-jnrsc\" (UID: \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\") " pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:13.363219 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.363064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fec8002a-2dce-4c5a-b601-5fdd2ac00461-cert\") pod \"kserve-controller-manager-575d6764f4-jnrsc\" (UID: \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\") " pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:13.365583 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.365561    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fec8002a-2dce-4c5a-b601-5fdd2ac00461-cert\") pod \"kserve-controller-manager-575d6764f4-jnrsc\" (UID: \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\") " pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:13.370691 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.370662    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgc8h\" (UniqueName: \"kubernetes.io/projected/fec8002a-2dce-4c5a-b601-5fdd2ac00461-kube-api-access-pgc8h\") pod \"kserve-controller-manager-575d6764f4-jnrsc\" (UID: \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\") " pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:13.504039 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.503916    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:13.774351 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.774325    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-jnrsc"]
Sep 08 13:26:13.777286 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:26:13.777256    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec8002a_2dce_4c5a_b601_5fdd2ac00461.slice/crio-60bf5bbf0ab9043f9ff50c8fa5f7a67bee5897be26cf676638ef838ec6c4d578 WatchSource:0}: Error finding container 60bf5bbf0ab9043f9ff50c8fa5f7a67bee5897be26cf676638ef838ec6c4d578: Status 404 returned error can't find the container with id 60bf5bbf0ab9043f9ff50c8fa5f7a67bee5897be26cf676638ef838ec6c4d578
Sep 08 13:26:13.922104 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:13.922064    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc" event={"ID":"fec8002a-2dce-4c5a-b601-5fdd2ac00461","Type":"ContainerStarted","Data":"60bf5bbf0ab9043f9ff50c8fa5f7a67bee5897be26cf676638ef838ec6c4d578"}
Sep 08 13:26:15.228361 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:15.228322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:15.371997 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:15.371962    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:26:18.945699 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:18.945666    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc" event={"ID":"fec8002a-2dce-4c5a-b601-5fdd2ac00461","Type":"ContainerStarted","Data":"b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f"}
Sep 08 13:26:18.946152 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:18.945798    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:18.965573 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:18.965523    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc" podStartSLOduration=1.609191315 podStartE2EDuration="5.965506949s" totalImagesPullingTime="4.356315634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:26:18.963355832 +0000 UTC m=+607.892259579" watchObservedRunningTime="2026-09-08 13:26:18.965506949 +0000 UTC m=+607.894410690"
Sep 08 13:26:49.955336 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:49.955262    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:58.791071 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:58.791034    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-jnrsc"]
Sep 08 13:26:58.791501 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:58.791250    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc" podUID="fec8002a-2dce-4c5a-b601-5fdd2ac00461" containerName="manager" containerID="cri-o://b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f" gracePeriod=10
Sep 08 13:26:58.817364 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:58.817331    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-7tqb9"]
Sep 08 13:26:58.821523 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:58.821504    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:26:58.830343 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:58.830311    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-7tqb9"]
Sep 08 13:26:58.959213 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:58.959175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w2zc\" (UniqueName: \"kubernetes.io/projected/952814a1-83ac-44c3-9ca5-75fa210b9f36-kube-api-access-7w2zc\") pod \"kserve-controller-manager-575d6764f4-7tqb9\" (UID: \"952814a1-83ac-44c3-9ca5-75fa210b9f36\") " pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:26:58.959399 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:58.959245    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/952814a1-83ac-44c3-9ca5-75fa210b9f36-cert\") pod \"kserve-controller-manager-575d6764f4-7tqb9\" (UID: \"952814a1-83ac-44c3-9ca5-75fa210b9f36\") " pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:26:59.034174 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.034151    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:59.060506 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.060429    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w2zc\" (UniqueName: \"kubernetes.io/projected/952814a1-83ac-44c3-9ca5-75fa210b9f36-kube-api-access-7w2zc\") pod \"kserve-controller-manager-575d6764f4-7tqb9\" (UID: \"952814a1-83ac-44c3-9ca5-75fa210b9f36\") " pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:26:59.060670 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.060516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/952814a1-83ac-44c3-9ca5-75fa210b9f36-cert\") pod \"kserve-controller-manager-575d6764f4-7tqb9\" (UID: \"952814a1-83ac-44c3-9ca5-75fa210b9f36\") " pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:26:59.063363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.063336    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/952814a1-83ac-44c3-9ca5-75fa210b9f36-cert\") pod \"kserve-controller-manager-575d6764f4-7tqb9\" (UID: \"952814a1-83ac-44c3-9ca5-75fa210b9f36\") " pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:26:59.069186 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.069163    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w2zc\" (UniqueName: \"kubernetes.io/projected/952814a1-83ac-44c3-9ca5-75fa210b9f36-kube-api-access-7w2zc\") pod \"kserve-controller-manager-575d6764f4-7tqb9\" (UID: \"952814a1-83ac-44c3-9ca5-75fa210b9f36\") " pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:26:59.121815 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.121784    2583 generic.go:356] "Generic (PLEG): container finished" podID="fec8002a-2dce-4c5a-b601-5fdd2ac00461" containerID="b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f" exitCode=0
Sep 08 13:26:59.121957 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.121831    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc"
Sep 08 13:26:59.121957 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.121861    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc" event={"ID":"fec8002a-2dce-4c5a-b601-5fdd2ac00461","Type":"ContainerDied","Data":"b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f"}
Sep 08 13:26:59.121957 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.121929    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-jnrsc" event={"ID":"fec8002a-2dce-4c5a-b601-5fdd2ac00461","Type":"ContainerDied","Data":"60bf5bbf0ab9043f9ff50c8fa5f7a67bee5897be26cf676638ef838ec6c4d578"}
Sep 08 13:26:59.121957 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.121956    2583 scope.go:117] "RemoveContainer" containerID="b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f"
Sep 08 13:26:59.130246 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.130230    2583 scope.go:117] "RemoveContainer" containerID="b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f"
Sep 08 13:26:59.130531 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:26:59.130508    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f\": container with ID starting with b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f not found: ID does not exist" containerID="b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f"
Sep 08 13:26:59.130624 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.130540    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f"} err="failed to get container status \"b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f\": rpc error: code = NotFound desc = could not find container \"b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f\": container with ID starting with b13ffbe3925134dc7033b48bc0e40ffc2d77a0cd029b5e92e669d9c8777cfc5f not found: ID does not exist"
Sep 08 13:26:59.160855 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.160823    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fec8002a-2dce-4c5a-b601-5fdd2ac00461-cert\") pod \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\" (UID: \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\") "
Sep 08 13:26:59.160990 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.160894    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgc8h\" (UniqueName: \"kubernetes.io/projected/fec8002a-2dce-4c5a-b601-5fdd2ac00461-kube-api-access-pgc8h\") pod \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\" (UID: \"fec8002a-2dce-4c5a-b601-5fdd2ac00461\") "
Sep 08 13:26:59.163253 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.163226    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec8002a-2dce-4c5a-b601-5fdd2ac00461-kube-api-access-pgc8h" (OuterVolumeSpecName: "kube-api-access-pgc8h") pod "fec8002a-2dce-4c5a-b601-5fdd2ac00461" (UID: "fec8002a-2dce-4c5a-b601-5fdd2ac00461"). InnerVolumeSpecName "kube-api-access-pgc8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:26:59.163253 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.163238    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec8002a-2dce-4c5a-b601-5fdd2ac00461-cert" (OuterVolumeSpecName: "cert") pod "fec8002a-2dce-4c5a-b601-5fdd2ac00461" (UID: "fec8002a-2dce-4c5a-b601-5fdd2ac00461"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:26:59.167488 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.167470    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:26:59.261654 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.261608    2583 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fec8002a-2dce-4c5a-b601-5fdd2ac00461-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:26:59.261654 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.261645    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgc8h\" (UniqueName: \"kubernetes.io/projected/fec8002a-2dce-4c5a-b601-5fdd2ac00461-kube-api-access-pgc8h\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:26:59.443433 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.443398    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-jnrsc"]
Sep 08 13:26:59.446809 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.446780    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-7tqb9"]
Sep 08 13:26:59.449779 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.449756    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-575d6764f4-jnrsc"]
Sep 08 13:26:59.450483 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:26:59.450461    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952814a1_83ac_44c3_9ca5_75fa210b9f36.slice/crio-f850ab070f1d476dd2797a575e428469a34a370f82c1b9935f2cd8f96b406f92 WatchSource:0}: Error finding container f850ab070f1d476dd2797a575e428469a34a370f82c1b9935f2cd8f96b406f92: Status 404 returned error can't find the container with id f850ab070f1d476dd2797a575e428469a34a370f82c1b9935f2cd8f96b406f92
Sep 08 13:26:59.595307 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:26:59.595264    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fec8002a-2dce-4c5a-b601-5fdd2ac00461" path="/var/lib/kubelet/pods/fec8002a-2dce-4c5a-b601-5fdd2ac00461/volumes"
Sep 08 13:27:00.128097 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:00.128059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-7tqb9" event={"ID":"952814a1-83ac-44c3-9ca5-75fa210b9f36","Type":"ContainerStarted","Data":"d09c3b418eab577f5a295cc73e4cd9d6a3a5ed633ad89efc236037452910cce2"}
Sep 08 13:27:00.128588 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:00.128130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:27:00.128588 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:00.128152    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-575d6764f4-7tqb9" event={"ID":"952814a1-83ac-44c3-9ca5-75fa210b9f36","Type":"ContainerStarted","Data":"f850ab070f1d476dd2797a575e428469a34a370f82c1b9935f2cd8f96b406f92"}
Sep 08 13:27:00.151632 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:00.148745    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-575d6764f4-7tqb9" podStartSLOduration=1.709420655 podStartE2EDuration="2.148719833s" totalImagesPullingTime="439.299178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:00.14494783 +0000 UTC m=+649.073851572" watchObservedRunningTime="2026-09-08 13:27:00.148719833 +0000 UTC m=+649.077623575"
Sep 08 13:27:31.137345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:31.137314    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-575d6764f4-7tqb9"
Sep 08 13:27:32.483507 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.483474    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:32.552106 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.552071    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-99gt9"]
Sep 08 13:27:32.552649 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.552630    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fec8002a-2dce-4c5a-b601-5fdd2ac00461" containerName="manager"
Sep 08 13:27:32.552649 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.552649    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fec8002a-2dce-4c5a-b601-5fdd2ac00461" containerName="manager"
Sep 08 13:27:32.552803 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.552717    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fec8002a-2dce-4c5a-b601-5fdd2ac00461" containerName="manager"
Sep 08 13:27:32.557310 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.557292    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:32.560250 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.560226    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 08 13:27:32.560870 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.560847    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-cr8qf\""
Sep 08 13:27:32.561164 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.561148    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 08 13:27:32.570656 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.570624    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-99gt9"]
Sep 08 13:27:32.583183 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.583153    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:32.641432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.641396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc99e657-db72-42b1-a493-fd0e2c93a2fe-metrics-certs\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:32.641626 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.641481    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxdh2\" (UniqueName: \"kubernetes.io/projected/fc99e657-db72-42b1-a493-fd0e2c93a2fe-kube-api-access-qxdh2\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:32.641626 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.641565    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc99e657-db72-42b1-a493-fd0e2c93a2fe-cert\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:32.656232 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.656204    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:32.743069 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.742966    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc99e657-db72-42b1-a493-fd0e2c93a2fe-cert\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:32.743069 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.743039    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc99e657-db72-42b1-a493-fd0e2c93a2fe-metrics-certs\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:32.743310 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.743075    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxdh2\" (UniqueName: \"kubernetes.io/projected/fc99e657-db72-42b1-a493-fd0e2c93a2fe-kube-api-access-qxdh2\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:32.743310 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:27:32.743121    2583 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 08 13:27:32.743310 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:27:32.743196    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc99e657-db72-42b1-a493-fd0e2c93a2fe-cert podName:fc99e657-db72-42b1-a493-fd0e2c93a2fe nodeName:}" failed. No retries permitted until 2026-09-08 13:27:33.243176815 +0000 UTC m=+682.172080536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fc99e657-db72-42b1-a493-fd0e2c93a2fe-cert") pod "odh-model-controller-7b966d7984-99gt9" (UID: "fc99e657-db72-42b1-a493-fd0e2c93a2fe") : secret "odh-model-controller-webhook-cert" not found
Sep 08 13:27:32.745690 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.745664    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc99e657-db72-42b1-a493-fd0e2c93a2fe-metrics-certs\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:32.755930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.755895    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:32.756155 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:32.756125    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxdh2\" (UniqueName: \"kubernetes.io/projected/fc99e657-db72-42b1-a493-fd0e2c93a2fe-kube-api-access-qxdh2\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:33.013631 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:33.013546    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:33.247535 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:33.247505    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc99e657-db72-42b1-a493-fd0e2c93a2fe-cert\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:33.250100 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:33.250071    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc99e657-db72-42b1-a493-fd0e2c93a2fe-cert\") pod \"odh-model-controller-7b966d7984-99gt9\" (UID: \"fc99e657-db72-42b1-a493-fd0e2c93a2fe\") " pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:33.470418 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:33.470377    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:33.742612 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:33.742541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-99gt9"]
Sep 08 13:27:33.747859 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:27:33.747821    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc99e657_db72_42b1_a493_fd0e2c93a2fe.slice/crio-caf73a7d51dfc615715364fc4dffdeb77e7316fe3ea2a574b2fa524c1e21c7f0 WatchSource:0}: Error finding container caf73a7d51dfc615715364fc4dffdeb77e7316fe3ea2a574b2fa524c1e21c7f0: Status 404 returned error can't find the container with id caf73a7d51dfc615715364fc4dffdeb77e7316fe3ea2a574b2fa524c1e21c7f0
Sep 08 13:27:33.824218 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:33.824187    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:33.924581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:33.924541    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:27:34.271674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:34.271635    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-99gt9" event={"ID":"fc99e657-db72-42b1-a493-fd0e2c93a2fe","Type":"ContainerStarted","Data":"caf73a7d51dfc615715364fc4dffdeb77e7316fe3ea2a574b2fa524c1e21c7f0"}
Sep 08 13:27:37.288861 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:37.288821    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-99gt9" event={"ID":"fc99e657-db72-42b1-a493-fd0e2c93a2fe","Type":"ContainerStarted","Data":"c8c9abf6d3c1918e76e104833a2c42da7ceb49a0cd66b50d2bb6bf12f2aa5999"}
Sep 08 13:27:37.289394 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:37.288946    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:27:37.306718 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:37.306662    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-99gt9" podStartSLOduration=2.307921483 podStartE2EDuration="5.306644876s" totalImagesPullingTime="2.998723393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:27:37.305740425 +0000 UTC m=+686.234644171" watchObservedRunningTime="2026-09-08 13:27:37.306644876 +0000 UTC m=+686.235548717"
Sep 08 13:27:48.294504 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:27:48.294475    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-99gt9"
Sep 08 13:29:11.408532 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:11.408501    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:15.040182 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:15.040145    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:16.693408 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.693372    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-whg7m"]
Sep 08 13:29:16.696287 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.696259    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:16.698792 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.698771    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4vj2j\""
Sep 08 13:29:16.698932 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.698771    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 08 13:29:16.698932 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.698771    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 08 13:29:16.705102 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.705074    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-whg7m"]
Sep 08 13:29:16.758349 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.758311    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/f425f0b6-8c98-47fc-b79f-881748983b61-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-whg7m\" (UID: \"f425f0b6-8c98-47fc-b79f-881748983b61\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:16.758349 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.758354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4qjf\" (UniqueName: \"kubernetes.io/projected/f425f0b6-8c98-47fc-b79f-881748983b61-kube-api-access-f4qjf\") pod \"seaweedfs-tls-custom-8b44d767d-whg7m\" (UID: \"f425f0b6-8c98-47fc-b79f-881748983b61\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:16.758581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.758409    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f425f0b6-8c98-47fc-b79f-881748983b61-data\") pod \"seaweedfs-tls-custom-8b44d767d-whg7m\" (UID: \"f425f0b6-8c98-47fc-b79f-881748983b61\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:16.859429 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.859397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/f425f0b6-8c98-47fc-b79f-881748983b61-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-whg7m\" (UID: \"f425f0b6-8c98-47fc-b79f-881748983b61\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:16.859624 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.859439    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4qjf\" (UniqueName: \"kubernetes.io/projected/f425f0b6-8c98-47fc-b79f-881748983b61-kube-api-access-f4qjf\") pod \"seaweedfs-tls-custom-8b44d767d-whg7m\" (UID: \"f425f0b6-8c98-47fc-b79f-881748983b61\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:16.859624 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.859468    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f425f0b6-8c98-47fc-b79f-881748983b61-data\") pod \"seaweedfs-tls-custom-8b44d767d-whg7m\" (UID: \"f425f0b6-8c98-47fc-b79f-881748983b61\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:16.859882 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.859850    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f425f0b6-8c98-47fc-b79f-881748983b61-data\") pod \"seaweedfs-tls-custom-8b44d767d-whg7m\" (UID: \"f425f0b6-8c98-47fc-b79f-881748983b61\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:16.861967 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.861946    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/f425f0b6-8c98-47fc-b79f-881748983b61-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-whg7m\" (UID: \"f425f0b6-8c98-47fc-b79f-881748983b61\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:16.868126 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:16.868089    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4qjf\" (UniqueName: \"kubernetes.io/projected/f425f0b6-8c98-47fc-b79f-881748983b61-kube-api-access-f4qjf\") pod \"seaweedfs-tls-custom-8b44d767d-whg7m\" (UID: \"f425f0b6-8c98-47fc-b79f-881748983b61\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:17.006521 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:17.006436    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m"
Sep 08 13:29:17.289165 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:17.289043    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-whg7m"]
Sep 08 13:29:17.291779 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:29:17.291751    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf425f0b6_8c98_47fc_b79f_881748983b61.slice/crio-d0f8e0bcd54179564e7fb807bb2bbaf5c611ad7fccbc498519ef3f3b8930336b WatchSource:0}: Error finding container d0f8e0bcd54179564e7fb807bb2bbaf5c611ad7fccbc498519ef3f3b8930336b: Status 404 returned error can't find the container with id d0f8e0bcd54179564e7fb807bb2bbaf5c611ad7fccbc498519ef3f3b8930336b
Sep 08 13:29:17.729752 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:17.729711    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m" event={"ID":"f425f0b6-8c98-47fc-b79f-881748983b61","Type":"ContainerStarted","Data":"d0f8e0bcd54179564e7fb807bb2bbaf5c611ad7fccbc498519ef3f3b8930336b"}
Sep 08 13:29:20.744991 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:20.744950    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m" event={"ID":"f425f0b6-8c98-47fc-b79f-881748983b61","Type":"ContainerStarted","Data":"9b93dfb4ed99e48e232b11c8d2e877c4c9a59044fc6e2d7bc4f5d495207c94e3"}
Sep 08 13:29:20.761440 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:20.761380    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-whg7m" podStartSLOduration=2.004168981 podStartE2EDuration="4.761359428s" totalImagesPullingTime="2.757190447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:29:20.760690459 +0000 UTC m=+789.689594223" watchObservedRunningTime="2026-09-08 13:29:20.761359428 +0000 UTC m=+789.690263171"
Sep 08 13:29:33.350065 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:33.349948    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:39.565650 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:39.565614    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:46.238151 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:46.238116    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:46.337946 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:46.337913    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:57.911907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:57.911868    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:58.118493 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:58.118455    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:29:58.654683 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:29:58.654645    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:01.284555 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:30:01.284519    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:02.982477 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:30:02.982442    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:08.123466 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:30:08.123433    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:08.270353 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:30:08.270322    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:34.373370 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:30:34.373335    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:30:49.057036 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:30:49.056975    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:11.559241 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:31:11.559207    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:31:11.560569 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:31:11.560546    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:31:53.044203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:31:53.044150    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:53.122288 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:31:53.122253    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:31:53.187849 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:31:53.187818    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:08.248097 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:32:08.248067    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:13.397186 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:32:13.397150    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:13.482127 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:32:13.482093    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:32:14.852513 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:32:14.852477    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:28.181057 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:28.180997    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:28.267398 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:28.267363    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:28.439479 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:28.439398    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:43.514138 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:43.514102    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:48.294142 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.289270    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:48.660116 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.660081    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:48.688320 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.688278    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"]
Sep 08 13:33:48.691991 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.691969    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.694418 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.694394    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 08 13:33:48.694553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.694459    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 08 13:33:48.694553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.694540    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:33:48.694553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.694459    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:33:48.694699 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.694648    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:33:48.694968 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.694952    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:33:48.695055 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.694981    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:33:48.703275 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.703248    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"]
Sep 08 13:33:48.735108 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.735079    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:48.792151 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.792116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.792343 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.792169    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d8jt\" (UniqueName: \"kubernetes.io/projected/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-api-access-5d8jt\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.792343 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.792283    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d779974-c26c-4aac-86c2-60c7039bb8c1-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.792437 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.792360    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.792437 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.792402    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d779974-c26c-4aac-86c2-60c7039bb8c1-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.893382 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.893342    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.893566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.893399    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d8jt\" (UniqueName: \"kubernetes.io/projected/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-api-access-5d8jt\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.893566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.893444    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d779974-c26c-4aac-86c2-60c7039bb8c1-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.893566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.893480    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.893566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.893519    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d779974-c26c-4aac-86c2-60c7039bb8c1-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.893764 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:33:48.893603    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 08 13:33:48.893764 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:33:48.893678    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d779974-c26c-4aac-86c2-60c7039bb8c1-proxy-tls podName:0d779974-c26c-4aac-86c2-60c7039bb8c1 nodeName:}" failed. No retries permitted until 2026-09-08 13:33:49.393653981 +0000 UTC m=+1058.322557703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d779974-c26c-4aac-86c2-60c7039bb8c1-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" (UID: "0d779974-c26c-4aac-86c2-60c7039bb8c1") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 08 13:33:48.893933 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.893917    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d779974-c26c-4aac-86c2-60c7039bb8c1-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.894189 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.894173    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.894235 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.894203    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:48.902089 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:48.902065    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d8jt\" (UniqueName: \"kubernetes.io/projected/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-api-access-5d8jt\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:49.158498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:49.158466    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:33:49.397592 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:49.397554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d779974-c26c-4aac-86c2-60c7039bb8c1-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:49.400311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:49.400285    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d779974-c26c-4aac-86c2-60c7039bb8c1-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:49.605774 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:49.605731    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:33:49.883513 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:49.883433    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"]
Sep 08 13:33:49.886920 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:33:49.886892    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d779974_c26c_4aac_86c2_60c7039bb8c1.slice/crio-a3f8e6398fab2ce834b8de9ea8dbe16e952dea6b65cdfd526e81c45a58e08915 WatchSource:0}: Error finding container a3f8e6398fab2ce834b8de9ea8dbe16e952dea6b65cdfd526e81c45a58e08915: Status 404 returned error can't find the container with id a3f8e6398fab2ce834b8de9ea8dbe16e952dea6b65cdfd526e81c45a58e08915
Sep 08 13:33:49.888956 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:49.888939    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:33:49.942312 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:49.942274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" event={"ID":"0d779974-c26c-4aac-86c2-60c7039bb8c1","Type":"ContainerStarted","Data":"a3f8e6398fab2ce834b8de9ea8dbe16e952dea6b65cdfd526e81c45a58e08915"}
Sep 08 13:33:55.978240 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:33:55.978196    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" event={"ID":"0d779974-c26c-4aac-86c2-60c7039bb8c1","Type":"ContainerStarted","Data":"b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb"}
Sep 08 13:34:01.003761 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:01.003726    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerID="b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb" exitCode=0
Sep 08 13:34:01.004146 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:01.003802    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" event={"ID":"0d779974-c26c-4aac-86c2-60c7039bb8c1","Type":"ContainerDied","Data":"b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb"}
Sep 08 13:34:23.139768 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:23.139731    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" event={"ID":"0d779974-c26c-4aac-86c2-60c7039bb8c1","Type":"ContainerStarted","Data":"378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3"}
Sep 08 13:34:25.152955 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:25.152911    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" event={"ID":"0d779974-c26c-4aac-86c2-60c7039bb8c1","Type":"ContainerStarted","Data":"c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a"}
Sep 08 13:34:25.153369 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:25.153052    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:34:25.177362 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:25.177305    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podStartSLOduration=3.41178836 podStartE2EDuration="37.177288919s" totalImagesPullingTime="29.701546186s" totalInitContainerRuntime="4.063954373s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:33:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:34:25.174517888 +0000 UTC m=+1094.103421641" watchObservedRunningTime="2026-09-08 13:34:25.177288919 +0000 UTC m=+1094.106192660"
Sep 08 13:34:26.157894 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:26.157869    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:34:26.159329 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:26.159294    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:34:27.161818 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:27.161769    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:34:32.166144 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:32.166114    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:34:32.166685 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:32.166659    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:34:42.166841 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:42.166803    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:34:52.167255 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:34:52.167214    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:35:02.166663 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:02.166621    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:35:12.166834 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:12.166790    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:35:22.167155 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:22.167114    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:35:23.630251 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:23.630210    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:27.253910 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:27.253867    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:32.167582 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:32.167544    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:35:39.238258 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:39.238223    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:42.168126 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:42.168097    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:35:49.030363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:49.030329    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"]
Sep 08 13:35:49.030848 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:49.030575    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" containerID="cri-o://378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3" gracePeriod=30
Sep 08 13:35:49.030848 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:49.030601    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kube-rbac-proxy" containerID="cri-o://c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a" gracePeriod=30
Sep 08 13:35:49.084160 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:49.084112    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:49.543497 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:49.543464    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerID="c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a" exitCode=2
Sep 08 13:35:49.543678 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:49.543525    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" event={"ID":"0d779974-c26c-4aac-86c2-60c7039bb8c1","Type":"ContainerDied","Data":"c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a"}
Sep 08 13:35:52.162352 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:52.162307    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Sep 08 13:35:52.166787 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:52.166750    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Sep 08 13:35:54.255430 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.255382    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:54.384435 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.384394    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"]
Sep 08 13:35:54.388414 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.388392    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.392553 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:54.392516    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-137-66.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-137-66.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 08 13:35:54.392553 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:54.392533    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" is forbidden: User \"system:node:ip-10-0-137-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-137-66.ec2.internal' and this object" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.392726 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:54.392570    2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-predictive-sklearn-v2-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-137-66.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-137-66.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\"" type="*v1.Secret"
Sep 08 13:35:54.420974 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.420937    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"]
Sep 08 13:35:54.421395 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.421364    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.421505 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.421451    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.421505 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.421492    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.421623 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.421543    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnx9s\" (UniqueName: \"kubernetes.io/projected/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-api-access-nnx9s\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.421720 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.421694    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.486947 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.486922    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:35:54.489381 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.489352    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:35:54.522138 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522053    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-rbac-proxy-sar-config\") pod \"0d779974-c26c-4aac-86c2-60c7039bb8c1\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") "
Sep 08 13:35:54.522138 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522135    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5d8jt\" (UniqueName: \"kubernetes.io/projected/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-api-access-5d8jt\") pod \"0d779974-c26c-4aac-86c2-60c7039bb8c1\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") "
Sep 08 13:35:54.522368 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522163    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d779974-c26c-4aac-86c2-60c7039bb8c1-kserve-provision-location\") pod \"0d779974-c26c-4aac-86c2-60c7039bb8c1\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") "
Sep 08 13:35:54.522368 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522213    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-cabundle-cert\") pod \"0d779974-c26c-4aac-86c2-60c7039bb8c1\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") "
Sep 08 13:35:54.522368 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522229    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d779974-c26c-4aac-86c2-60c7039bb8c1-proxy-tls\") pod \"0d779974-c26c-4aac-86c2-60c7039bb8c1\" (UID: \"0d779974-c26c-4aac-86c2-60c7039bb8c1\") "
Sep 08 13:35:54.522368 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnx9s\" (UniqueName: \"kubernetes.io/projected/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-api-access-nnx9s\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.522579 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522390    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.522579 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522453    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d779974-c26c-4aac-86c2-60c7039bb8c1" (UID: "0d779974-c26c-4aac-86c2-60c7039bb8c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:54.522579 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.522728 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522639    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d779974-c26c-4aac-86c2-60c7039bb8c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d779974-c26c-4aac-86c2-60c7039bb8c1" (UID: "0d779974-c26c-4aac-86c2-60c7039bb8c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:35:54.522728 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522654    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d779974-c26c-4aac-86c2-60c7039bb8c1" (UID: "0d779974-c26c-4aac-86c2-60c7039bb8c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:35:54.522728 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522669    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.522728 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.523114 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522764    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:35:54.523114 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522778    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:35:54.523114 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522787    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d779974-c26c-4aac-86c2-60c7039bb8c1-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:35:54.523114 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.522987    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.523322 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.523293    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.524735 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.524711    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-api-access-5d8jt" (OuterVolumeSpecName: "kube-api-access-5d8jt") pod "0d779974-c26c-4aac-86c2-60c7039bb8c1" (UID: "0d779974-c26c-4aac-86c2-60c7039bb8c1"). InnerVolumeSpecName "kube-api-access-5d8jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:35:54.524735 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.524711    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d779974-c26c-4aac-86c2-60c7039bb8c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d779974-c26c-4aac-86c2-60c7039bb8c1" (UID: "0d779974-c26c-4aac-86c2-60c7039bb8c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:35:54.537858 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.537832    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnx9s\" (UniqueName: \"kubernetes.io/projected/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-api-access-nnx9s\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:54.567593 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.567547    2583 generic.go:356] "Generic (PLEG): container finished" podID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerID="378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3" exitCode=0
Sep 08 13:35:54.567748 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.567621    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" event={"ID":"0d779974-c26c-4aac-86c2-60c7039bb8c1","Type":"ContainerDied","Data":"378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3"}
Sep 08 13:35:54.567748 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.567635    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"
Sep 08 13:35:54.567748 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.567681    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5" event={"ID":"0d779974-c26c-4aac-86c2-60c7039bb8c1","Type":"ContainerDied","Data":"a3f8e6398fab2ce834b8de9ea8dbe16e952dea6b65cdfd526e81c45a58e08915"}
Sep 08 13:35:54.567748 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.567709    2583 scope.go:117] "RemoveContainer" containerID="c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a"
Sep 08 13:35:54.579156 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.579053    2583 scope.go:117] "RemoveContainer" containerID="378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3"
Sep 08 13:35:54.588100 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.588073    2583 scope.go:117] "RemoveContainer" containerID="b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb"
Sep 08 13:35:54.596572 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.596552    2583 scope.go:117] "RemoveContainer" containerID="c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a"
Sep 08 13:35:54.596897 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:54.596874    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a\": container with ID starting with c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a not found: ID does not exist" containerID="c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a"
Sep 08 13:35:54.596972 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.596906    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a"} err="failed to get container status \"c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a\": rpc error: code = NotFound desc = could not find container \"c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a\": container with ID starting with c9f0ee969d392c3911dc459e42b10515b7d40668cdcd33d543b5ac87b2d2db5a not found: ID does not exist"
Sep 08 13:35:54.596972 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.596925    2583 scope.go:117] "RemoveContainer" containerID="378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3"
Sep 08 13:35:54.597245 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:54.597223    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3\": container with ID starting with 378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3 not found: ID does not exist" containerID="378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3"
Sep 08 13:35:54.597292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.597250    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3"} err="failed to get container status \"378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3\": rpc error: code = NotFound desc = could not find container \"378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3\": container with ID starting with 378f065588a8f1a315a0d80d4ffdb853c9d52b80fab881cecd7573adf249dea3 not found: ID does not exist"
Sep 08 13:35:54.597292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.597266    2583 scope.go:117] "RemoveContainer" containerID="b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb"
Sep 08 13:35:54.597531 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:54.597514    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb\": container with ID starting with b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb not found: ID does not exist" containerID="b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb"
Sep 08 13:35:54.597587 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.597540    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb"} err="failed to get container status \"b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb\": rpc error: code = NotFound desc = could not find container \"b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb\": container with ID starting with b78a9ef252642a17081ea3acdd0d20d750ea26f4cf312c51587848ce7fe1efeb not found: ID does not exist"
Sep 08 13:35:54.601832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.601809    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"]
Sep 08 13:35:54.606732 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.606710    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-fr2n5"]
Sep 08 13:35:54.623700 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.623671    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d779974-c26c-4aac-86c2-60c7039bb8c1-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:35:54.623700 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:54.623695    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d8jt\" (UniqueName: \"kubernetes.io/projected/0d779974-c26c-4aac-86c2-60c7039bb8c1-kube-api-access-5d8jt\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:35:55.523518 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:55.523481    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 08 13:35:55.523915 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:55.523577    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-proxy-tls podName:abbd21a7-bc1f-44b2-a584-bc297ffabf2d nodeName:}" failed. No retries permitted until 2026-09-08 13:35:56.023559819 +0000 UTC m=+1184.952463542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" (UID: "abbd21a7-bc1f-44b2-a584-bc297ffabf2d") : failed to sync secret cache: timed out waiting for the condition
Sep 08 13:35:55.523915 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:55.523487    2583 configmap.go:193] Couldn't get configMap kserve-ci-e2e-test/isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config: failed to sync configmap cache: timed out waiting for the condition
Sep 08 13:35:55.523915 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:35:55.523660    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-rbac-proxy-sar-config podName:abbd21a7-bc1f-44b2-a584-bc297ffabf2d nodeName:}" failed. No retries permitted until 2026-09-08 13:35:56.023643949 +0000 UTC m=+1184.952547680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" (UniqueName: "kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-rbac-proxy-sar-config") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" (UID: "abbd21a7-bc1f-44b2-a584-bc297ffabf2d") : failed to sync configmap cache: timed out waiting for the condition
Sep 08 13:35:55.595162 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:55.595126    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" path="/var/lib/kubelet/pods/0d779974-c26c-4aac-86c2-60c7039bb8c1/volumes"
Sep 08 13:35:55.658272 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:55.658246    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 08 13:35:55.676701 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:55.676670    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 08 13:35:56.036609 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:56.036573    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:56.036784 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:56.036622    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:56.037323 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:56.037304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:56.039273 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:56.039255    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:56.199993 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:56.199953    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:35:56.487638 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:56.487607    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"]
Sep 08 13:35:56.490818 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:35:56.490772    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbd21a7_bc1f_44b2_a584_bc297ffabf2d.slice/crio-43f74faaefa29ab7058644bca03977283f00dba7ac01dca24927969560602e4f WatchSource:0}: Error finding container 43f74faaefa29ab7058644bca03977283f00dba7ac01dca24927969560602e4f: Status 404 returned error can't find the container with id 43f74faaefa29ab7058644bca03977283f00dba7ac01dca24927969560602e4f
Sep 08 13:35:56.579125 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:56.579087    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" event={"ID":"abbd21a7-bc1f-44b2-a584-bc297ffabf2d","Type":"ContainerStarted","Data":"2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d"}
Sep 08 13:35:56.579500 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:35:56.579160    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" event={"ID":"abbd21a7-bc1f-44b2-a584-bc297ffabf2d","Type":"ContainerStarted","Data":"43f74faaefa29ab7058644bca03977283f00dba7ac01dca24927969560602e4f"}
Sep 08 13:36:00.598889 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:00.598854    2583 generic.go:356] "Generic (PLEG): container finished" podID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerID="2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d" exitCode=0
Sep 08 13:36:00.599299 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:00.598925    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" event={"ID":"abbd21a7-bc1f-44b2-a584-bc297ffabf2d","Type":"ContainerDied","Data":"2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d"}
Sep 08 13:36:01.605507 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:01.605474    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" event={"ID":"abbd21a7-bc1f-44b2-a584-bc297ffabf2d","Type":"ContainerStarted","Data":"926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5"}
Sep 08 13:36:01.605918 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:01.605544    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" event={"ID":"abbd21a7-bc1f-44b2-a584-bc297ffabf2d","Type":"ContainerStarted","Data":"1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda"}
Sep 08 13:36:01.605918 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:01.605739    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:36:01.630094 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:01.630039    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" podStartSLOduration=3.748863731 podStartE2EDuration="7.63000343s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.881139699s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:36:01.628927815 +0000 UTC m=+1190.557831585" watchObservedRunningTime="2026-09-08 13:36:01.63000343 +0000 UTC m=+1190.558907173"
Sep 08 13:36:02.610501 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:02.610468    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:36:08.620342 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:08.620314    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:36:09.006847 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:09.006754    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:36:11.596406 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:11.596368    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:36:11.599770 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:11.597496    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:36:38.621692 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:38.621652    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.29:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:36:48.620872 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:48.620811    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.29:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:36:58.621639 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:36:58.621595    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.29:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:37:08.620915 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:08.620832    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.29:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:37:09.388892 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:09.388850    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:17.615276 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:17.615243    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:37:24.344173 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:24.344136    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"]
Sep 08 13:37:24.344668 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:24.344373    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kserve-container" containerID="cri-o://1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda" gracePeriod=30
Sep 08 13:37:24.344668 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:24.344421    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kube-rbac-proxy" containerID="cri-o://926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5" gracePeriod=30
Sep 08 13:37:24.356690 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:24.356651    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:24.986344 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:24.986311    2583 generic.go:356] "Generic (PLEG): container finished" podID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerID="926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5" exitCode=2
Sep 08 13:37:24.986530 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:24.986390    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" event={"ID":"abbd21a7-bc1f-44b2-a584-bc297ffabf2d","Type":"ContainerDied","Data":"926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5"}
Sep 08 13:37:27.590705 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:27.590657    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.29:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 08 13:37:28.615502 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:28.615460    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 08 13:37:29.457742 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.457705    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:29.510229 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.510205    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:37:29.568725 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.568644    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"]
Sep 08 13:37:29.569061 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569047    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kube-rbac-proxy"
Sep 08 13:37:29.569113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569063    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kube-rbac-proxy"
Sep 08 13:37:29.569113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569071    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="storage-initializer"
Sep 08 13:37:29.569113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569077    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="storage-initializer"
Sep 08 13:37:29.569113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569085    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="storage-initializer"
Sep 08 13:37:29.569113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569090    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="storage-initializer"
Sep 08 13:37:29.569113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569098    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kube-rbac-proxy"
Sep 08 13:37:29.569113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569103    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kube-rbac-proxy"
Sep 08 13:37:29.569113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569109    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container"
Sep 08 13:37:29.569113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569114    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container"
Sep 08 13:37:29.569363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569120    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kserve-container"
Sep 08 13:37:29.569363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569125    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kserve-container"
Sep 08 13:37:29.569363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569178    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kube-rbac-proxy"
Sep 08 13:37:29.569363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569185    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kserve-container"
Sep 08 13:37:29.569363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569191    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerName="kserve-container"
Sep 08 13:37:29.569363 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.569198    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d779974-c26c-4aac-86c2-60c7039bb8c1" containerName="kube-rbac-proxy"
Sep 08 13:37:29.572284 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.572251    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.574836 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.574813    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 08 13:37:29.574970 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.574947    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 08 13:37:29.599061 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.599035    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"]
Sep 08 13:37:29.624551 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.624512    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-proxy-tls\") pod \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") "
Sep 08 13:37:29.624551 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.624546    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-cabundle-cert\") pod \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") "
Sep 08 13:37:29.625109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.624574    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnx9s\" (UniqueName: \"kubernetes.io/projected/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-api-access-nnx9s\") pod \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") "
Sep 08 13:37:29.625109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.624593    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kserve-provision-location\") pod \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") "
Sep 08 13:37:29.625109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.624612    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-rbac-proxy-sar-config\") pod \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\" (UID: \"abbd21a7-bc1f-44b2-a584-bc297ffabf2d\") "
Sep 08 13:37:29.625109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.624956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abbd21a7-bc1f-44b2-a584-bc297ffabf2d" (UID: "abbd21a7-bc1f-44b2-a584-bc297ffabf2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:37:29.625109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.624983    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "abbd21a7-bc1f-44b2-a584-bc297ffabf2d" (UID: "abbd21a7-bc1f-44b2-a584-bc297ffabf2d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:37:29.625109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.625039    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "abbd21a7-bc1f-44b2-a584-bc297ffabf2d" (UID: "abbd21a7-bc1f-44b2-a584-bc297ffabf2d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:37:29.626900 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.626880    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "abbd21a7-bc1f-44b2-a584-bc297ffabf2d" (UID: "abbd21a7-bc1f-44b2-a584-bc297ffabf2d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:37:29.627022 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.626988    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-api-access-nnx9s" (OuterVolumeSpecName: "kube-api-access-nnx9s") pod "abbd21a7-bc1f-44b2-a584-bc297ffabf2d" (UID: "abbd21a7-bc1f-44b2-a584-bc297ffabf2d"). InnerVolumeSpecName "kube-api-access-nnx9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:37:29.701580 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.701546    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:29.725606 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.725568    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.725606 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.725612    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.725869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.725638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sph92\" (UniqueName: \"kubernetes.io/projected/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-api-access-sph92\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.725869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.725716    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.725997 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.725960    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.726109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.726088    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:37:29.726109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.726110    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:37:29.726292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.726122    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnx9s\" (UniqueName: \"kubernetes.io/projected/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-api-access-nnx9s\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:37:29.726292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.726131    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:37:29.726292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.726141    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/abbd21a7-bc1f-44b2-a584-bc297ffabf2d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:37:29.827331 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.827229    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.827331 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.827286    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.827331 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.827309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.827622 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.827337    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sph92\" (UniqueName: \"kubernetes.io/projected/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-api-access-sph92\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.827622 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.827358    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.827728 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.827708    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.828052 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.828005    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.828163 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.828057    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.829995 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.829977    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.841557 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.841525    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sph92\" (UniqueName: \"kubernetes.io/projected/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-api-access-sph92\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:29.884298 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:29.884261    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:30.012935 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.012902    2583 generic.go:356] "Generic (PLEG): container finished" podID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" containerID="1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda" exitCode=0
Sep 08 13:37:30.013112 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.012960    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"
Sep 08 13:37:30.013112 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.012965    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" event={"ID":"abbd21a7-bc1f-44b2-a584-bc297ffabf2d","Type":"ContainerDied","Data":"1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda"}
Sep 08 13:37:30.013112 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.013041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w" event={"ID":"abbd21a7-bc1f-44b2-a584-bc297ffabf2d","Type":"ContainerDied","Data":"43f74faaefa29ab7058644bca03977283f00dba7ac01dca24927969560602e4f"}
Sep 08 13:37:30.013112 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.013068    2583 scope.go:117] "RemoveContainer" containerID="926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5"
Sep 08 13:37:30.022598 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.022579    2583 scope.go:117] "RemoveContainer" containerID="1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda"
Sep 08 13:37:30.030833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.030814    2583 scope.go:117] "RemoveContainer" containerID="2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d"
Sep 08 13:37:30.039358 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.039328    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"]
Sep 08 13:37:30.039629 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.039613    2583 scope.go:117] "RemoveContainer" containerID="926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5"
Sep 08 13:37:30.039972 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:37:30.039953    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5\": container with ID starting with 926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5 not found: ID does not exist" containerID="926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5"
Sep 08 13:37:30.040104 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.040000    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5"} err="failed to get container status \"926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5\": rpc error: code = NotFound desc = could not find container \"926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5\": container with ID starting with 926542cf23f80eef1f3243994dc241dce0c47d847293f6a72d3f69664aa802f5 not found: ID does not exist"
Sep 08 13:37:30.040238 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.040107    2583 scope.go:117] "RemoveContainer" containerID="1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda"
Sep 08 13:37:30.040444 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:37:30.040425    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda\": container with ID starting with 1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda not found: ID does not exist" containerID="1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda"
Sep 08 13:37:30.040493 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.040452    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda"} err="failed to get container status \"1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda\": rpc error: code = NotFound desc = could not find container \"1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda\": container with ID starting with 1efbf10d2a0d8c2e0321d99312a830ecaa2daabe0d230a086ef724ac5bd1beda not found: ID does not exist"
Sep 08 13:37:30.040493 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.040470    2583 scope.go:117] "RemoveContainer" containerID="2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d"
Sep 08 13:37:30.040716 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:37:30.040698    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d\": container with ID starting with 2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d not found: ID does not exist" containerID="2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d"
Sep 08 13:37:30.040769 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.040723    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d"} err="failed to get container status \"2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d\": rpc error: code = NotFound desc = could not find container \"2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d\": container with ID starting with 2e5f43a638882f9763441ce67e30d9065697307add1e906459a6d9186add6c7d not found: ID does not exist"
Sep 08 13:37:30.044584 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.044562    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-9sn7w"]
Sep 08 13:37:30.171871 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:30.171845    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"]
Sep 08 13:37:30.173532 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:37:30.173504    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d31303_b3e9_4f4a_8a8c_2ff72a6677b6.slice/crio-68d1584de0a048ca00c0c91be4db52fe0622fbc78363cc4f367d7487d1f86c3a WatchSource:0}: Error finding container 68d1584de0a048ca00c0c91be4db52fe0622fbc78363cc4f367d7487d1f86c3a: Status 404 returned error can't find the container with id 68d1584de0a048ca00c0c91be4db52fe0622fbc78363cc4f367d7487d1f86c3a
Sep 08 13:37:31.019942 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:31.019865    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" event={"ID":"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6","Type":"ContainerStarted","Data":"708139b63ead41f6b231fb16e48948fac3635a3ec3721dcc99a54c0e61f2118c"}
Sep 08 13:37:31.019942 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:31.019940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" event={"ID":"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6","Type":"ContainerStarted","Data":"68d1584de0a048ca00c0c91be4db52fe0622fbc78363cc4f367d7487d1f86c3a"}
Sep 08 13:37:31.595393 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:31.595353    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abbd21a7-bc1f-44b2-a584-bc297ffabf2d" path="/var/lib/kubelet/pods/abbd21a7-bc1f-44b2-a584-bc297ffabf2d/volumes"
Sep 08 13:37:35.039426 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:35.039387    2583 generic.go:356] "Generic (PLEG): container finished" podID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerID="708139b63ead41f6b231fb16e48948fac3635a3ec3721dcc99a54c0e61f2118c" exitCode=0
Sep 08 13:37:35.039824 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:35.039459    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" event={"ID":"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6","Type":"ContainerDied","Data":"708139b63ead41f6b231fb16e48948fac3635a3ec3721dcc99a54c0e61f2118c"}
Sep 08 13:37:36.046313 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:36.046277    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" event={"ID":"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6","Type":"ContainerStarted","Data":"2322cbc7c9c4b00d6fe6bf0d824135656563a1cef554595295d46b315047cc10"}
Sep 08 13:37:36.046702 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:36.046341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" event={"ID":"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6","Type":"ContainerStarted","Data":"bd09384417f4bb3fc6db11eddb4eda055762611b603fe10e48d6e691727cbb06"}
Sep 08 13:37:36.046702 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:36.046658    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:36.046783 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:36.046713    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:36.068619 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:36.068551    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" podStartSLOduration=3.176265363 podStartE2EDuration="7.068533298s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.892267935s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:37:36.067705544 +0000 UTC m=+1284.996609297" watchObservedRunningTime="2026-09-08 13:37:36.068533298 +0000 UTC m=+1284.997437041"
Sep 08 13:37:42.056417 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:42.056386    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:37:49.928055 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:49.927987    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:55.084778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:55.084743    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:37:55.341692 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:37:55.341603    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:38:12.058002 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:38:12.057913    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 08 13:38:22.057692 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:38:22.057653    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 08 13:38:32.057068 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:38:32.057000    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 08 13:38:42.057817 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:38:42.057770    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 08 13:38:44.590708 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:38:44.590658    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 08 13:38:54.616112 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:38:54.616069    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:38:59.596328 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:38:59.596293    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"]
Sep 08 13:38:59.596725 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:38:59.596523    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kserve-container" containerID="cri-o://bd09384417f4bb3fc6db11eddb4eda055762611b603fe10e48d6e691727cbb06" gracePeriod=30
Sep 08 13:38:59.596725 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:38:59.596555    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kube-rbac-proxy" containerID="cri-o://2322cbc7c9c4b00d6fe6bf0d824135656563a1cef554595295d46b315047cc10" gracePeriod=30
Sep 08 13:39:00.419389 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:00.419352    2583 generic.go:356] "Generic (PLEG): container finished" podID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerID="2322cbc7c9c4b00d6fe6bf0d824135656563a1cef554595295d46b315047cc10" exitCode=2
Sep 08 13:39:00.419639 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:00.419399    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" event={"ID":"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6","Type":"ContainerDied","Data":"2322cbc7c9c4b00d6fe6bf0d824135656563a1cef554595295d46b315047cc10"}
Sep 08 13:39:02.051385 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:02.051335    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 08 13:39:02.815485 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:02.815449    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:04.439304 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.439267    2583 generic.go:356] "Generic (PLEG): container finished" podID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerID="bd09384417f4bb3fc6db11eddb4eda055762611b603fe10e48d6e691727cbb06" exitCode=0
Sep 08 13:39:04.439779 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.439329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" event={"ID":"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6","Type":"ContainerDied","Data":"bd09384417f4bb3fc6db11eddb4eda055762611b603fe10e48d6e691727cbb06"}
Sep 08 13:39:04.545635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.545607    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:39:04.649701 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.649611    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-proxy-tls\") pod \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") "
Sep 08 13:39:04.649701 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.649663    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sph92\" (UniqueName: \"kubernetes.io/projected/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-api-access-sph92\") pod \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") "
Sep 08 13:39:04.649701 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.649691    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kserve-provision-location\") pod \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") "
Sep 08 13:39:04.649988 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.649866    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-rbac-proxy-sar-config\") pod \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") "
Sep 08 13:39:04.649988 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.649898    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-cabundle-cert\") pod \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\" (UID: \"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6\") "
Sep 08 13:39:04.650129 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.649986    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" (UID: "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:39:04.650221 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.650206    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:39:04.650347 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.650321    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" (UID: "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:39:04.650347 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.650330    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" (UID: "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:39:04.652301 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.652271    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-api-access-sph92" (OuterVolumeSpecName: "kube-api-access-sph92") pod "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" (UID: "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6"). InnerVolumeSpecName "kube-api-access-sph92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:39:04.652404 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.652300    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" (UID: "59d31303-b3e9-4f4a-8a8c-2ff72a6677b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:39:04.751337 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.751241    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:39:04.751337 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.751282    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sph92\" (UniqueName: \"kubernetes.io/projected/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-api-access-sph92\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:39:04.751337 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.751297    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:39:04.751337 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.751311    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:39:04.777514 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.777482    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:04.830352 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.830317    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"]
Sep 08 13:39:04.830750 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.830733    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kserve-container"
Sep 08 13:39:04.830833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.830753    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kserve-container"
Sep 08 13:39:04.830833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.830783    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kube-rbac-proxy"
Sep 08 13:39:04.830833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.830791    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kube-rbac-proxy"
Sep 08 13:39:04.830833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.830819    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="storage-initializer"
Sep 08 13:39:04.830833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.830831    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="storage-initializer"
Sep 08 13:39:04.831108 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.830908    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kube-rbac-proxy"
Sep 08 13:39:04.831108 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.830921    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" containerName="kserve-container"
Sep 08 13:39:04.834553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.834531    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:04.837040 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.836996    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 08 13:39:04.837371 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.837356    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 08 13:39:04.854424 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.854391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"]
Sep 08 13:39:04.952298 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.952218    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrpt\" (UniqueName: \"kubernetes.io/projected/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-api-access-9jrpt\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:04.952298 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.952281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:04.952515 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.952302    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:04.952515 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.952350    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:04.952515 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.952418    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:04.996675 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:04.996643    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:05.053093 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.053055    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrpt\" (UniqueName: \"kubernetes.io/projected/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-api-access-9jrpt\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.053286 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.053111    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.053286 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.053131    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.053286 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.053260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.053474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.053327    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.053698 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.053679    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.053954 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.053877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.053954 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.053877    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.055744 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.055728    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.062429 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.062407    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrpt\" (UniqueName: \"kubernetes.io/projected/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-api-access-9jrpt\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.145195 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.145149    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:05.423427 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.423301    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"]
Sep 08 13:39:05.426114 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:39:05.426085    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbefcaa_b854_4afd_9afb_9662e5c9dda5.slice/crio-fbf48fd3410f97d24abec77d4f1cc26ab531b3457c25d19aa5c597e123c8df7e WatchSource:0}: Error finding container fbf48fd3410f97d24abec77d4f1cc26ab531b3457c25d19aa5c597e123c8df7e: Status 404 returned error can't find the container with id fbf48fd3410f97d24abec77d4f1cc26ab531b3457c25d19aa5c597e123c8df7e
Sep 08 13:39:05.448212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.448179    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2" event={"ID":"59d31303-b3e9-4f4a-8a8c-2ff72a6677b6","Type":"ContainerDied","Data":"68d1584de0a048ca00c0c91be4db52fe0622fbc78363cc4f367d7487d1f86c3a"}
Sep 08 13:39:05.448590 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.448255    2583 scope.go:117] "RemoveContainer" containerID="2322cbc7c9c4b00d6fe6bf0d824135656563a1cef554595295d46b315047cc10"
Sep 08 13:39:05.448590 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.448200    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"
Sep 08 13:39:05.450679 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.450556    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" event={"ID":"fcbefcaa-b854-4afd-9afb-9662e5c9dda5","Type":"ContainerStarted","Data":"fbf48fd3410f97d24abec77d4f1cc26ab531b3457c25d19aa5c597e123c8df7e"}
Sep 08 13:39:05.463867 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.463843    2583 scope.go:117] "RemoveContainer" containerID="bd09384417f4bb3fc6db11eddb4eda055762611b603fe10e48d6e691727cbb06"
Sep 08 13:39:05.479990 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.479960    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"]
Sep 08 13:39:05.481004 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.480984    2583 scope.go:117] "RemoveContainer" containerID="708139b63ead41f6b231fb16e48948fac3635a3ec3721dcc99a54c0e61f2118c"
Sep 08 13:39:05.487193 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.487171    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-vcrz2"]
Sep 08 13:39:05.595769 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:05.595729    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d31303-b3e9-4f4a-8a8c-2ff72a6677b6" path="/var/lib/kubelet/pods/59d31303-b3e9-4f4a-8a8c-2ff72a6677b6/volumes"
Sep 08 13:39:06.457828 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:06.457793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" event={"ID":"fcbefcaa-b854-4afd-9afb-9662e5c9dda5","Type":"ContainerStarted","Data":"69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4"}
Sep 08 13:39:09.470937 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:09.470903    2583 generic.go:356] "Generic (PLEG): container finished" podID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerID="69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4" exitCode=0
Sep 08 13:39:09.471338 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:09.470976    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" event={"ID":"fcbefcaa-b854-4afd-9afb-9662e5c9dda5","Type":"ContainerDied","Data":"69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4"}
Sep 08 13:39:10.477212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:10.477179    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" event={"ID":"fcbefcaa-b854-4afd-9afb-9662e5c9dda5","Type":"ContainerStarted","Data":"5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7"}
Sep 08 13:39:10.477665 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:10.477238    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" event={"ID":"fcbefcaa-b854-4afd-9afb-9662e5c9dda5","Type":"ContainerStarted","Data":"6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb"}
Sep 08 13:39:10.477665 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:10.477446    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:10.477665 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:10.477499    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:10.503620 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:10.503571    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" podStartSLOduration=2.652704805 podStartE2EDuration="6.50355633s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.850851525s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:39:10.501491786 +0000 UTC m=+1379.430395539" watchObservedRunningTime="2026-09-08 13:39:10.50355633 +0000 UTC m=+1379.432460071"
Sep 08 13:39:16.488377 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:16.488347    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:39:46.490096 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:46.490054    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 08 13:39:51.247753 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:51.247720    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:56.489121 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:56.489075    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 08 13:39:58.073133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:58.073094    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:39:58.178832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:39:58.178796    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:06.489333 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:06.489240    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 08 13:40:16.489746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:16.489706    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 08 13:40:25.361822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:25.361779    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:26.516866 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:26.516825    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:40:34.892154 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:34.892122    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"]
Sep 08 13:40:34.892574 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:34.892351    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kserve-container" containerID="cri-o://6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb" gracePeriod=30
Sep 08 13:40:34.892574 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:34.892383    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kube-rbac-proxy" containerID="cri-o://5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7" gracePeriod=30
Sep 08 13:40:34.924921 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:34.924800    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:35.883362 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:35.883326    2583 generic.go:356] "Generic (PLEG): container finished" podID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerID="5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7" exitCode=2
Sep 08 13:40:35.883545 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:35.883396    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" event={"ID":"fcbefcaa-b854-4afd-9afb-9662e5c9dda5","Type":"ContainerDied","Data":"5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7"}
Sep 08 13:40:36.482693 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:36.482655    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 08 13:40:36.489663 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:36.489633    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 08 13:40:40.643432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.643407    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:40:40.698422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.698332    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kserve-provision-location\") pod \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") "
Sep 08 13:40:40.698422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.698398    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-rbac-proxy-sar-config\") pod \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") "
Sep 08 13:40:40.698422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.698417    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jrpt\" (UniqueName: \"kubernetes.io/projected/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-api-access-9jrpt\") pod \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") "
Sep 08 13:40:40.698705 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.698433    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-proxy-tls\") pod \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") "
Sep 08 13:40:40.698705 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.698473    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-cabundle-cert\") pod \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\" (UID: \"fcbefcaa-b854-4afd-9afb-9662e5c9dda5\") "
Sep 08 13:40:40.698848 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.698814    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fcbefcaa-b854-4afd-9afb-9662e5c9dda5" (UID: "fcbefcaa-b854-4afd-9afb-9662e5c9dda5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:40.698848 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.698813    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fcbefcaa-b854-4afd-9afb-9662e5c9dda5" (UID: "fcbefcaa-b854-4afd-9afb-9662e5c9dda5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:40:40.699000 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.698865    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fcbefcaa-b854-4afd-9afb-9662e5c9dda5" (UID: "fcbefcaa-b854-4afd-9afb-9662e5c9dda5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:40:40.700709 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.700676    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fcbefcaa-b854-4afd-9afb-9662e5c9dda5" (UID: "fcbefcaa-b854-4afd-9afb-9662e5c9dda5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:40:40.700709 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.700692    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-api-access-9jrpt" (OuterVolumeSpecName: "kube-api-access-9jrpt") pod "fcbefcaa-b854-4afd-9afb-9662e5c9dda5" (UID: "fcbefcaa-b854-4afd-9afb-9662e5c9dda5"). InnerVolumeSpecName "kube-api-access-9jrpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:40:40.799599 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.799560    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:40:40.799599 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.799593    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jrpt\" (UniqueName: \"kubernetes.io/projected/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kube-api-access-9jrpt\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:40:40.799599 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.799605    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:40:40.799851 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.799619    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:40:40.799851 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.799629    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fcbefcaa-b854-4afd-9afb-9662e5c9dda5-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:40:40.907223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.907181    2583 generic.go:356] "Generic (PLEG): container finished" podID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerID="6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb" exitCode=0
Sep 08 13:40:40.907414 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.907250    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"
Sep 08 13:40:40.907414 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.907273    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" event={"ID":"fcbefcaa-b854-4afd-9afb-9662e5c9dda5","Type":"ContainerDied","Data":"6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb"}
Sep 08 13:40:40.907414 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.907351    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx" event={"ID":"fcbefcaa-b854-4afd-9afb-9662e5c9dda5","Type":"ContainerDied","Data":"fbf48fd3410f97d24abec77d4f1cc26ab531b3457c25d19aa5c597e123c8df7e"}
Sep 08 13:40:40.907414 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.907391    2583 scope.go:117] "RemoveContainer" containerID="5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7"
Sep 08 13:40:40.916000 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.915982    2583 scope.go:117] "RemoveContainer" containerID="6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb"
Sep 08 13:40:40.924167 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.924147    2583 scope.go:117] "RemoveContainer" containerID="69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4"
Sep 08 13:40:40.931569 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.931543    2583 scope.go:117] "RemoveContainer" containerID="5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7"
Sep 08 13:40:40.931903 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:40:40.931884    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7\": container with ID starting with 5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7 not found: ID does not exist" containerID="5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7"
Sep 08 13:40:40.932001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.931918    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7"} err="failed to get container status \"5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7\": rpc error: code = NotFound desc = could not find container \"5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7\": container with ID starting with 5e48fd138943ccb4022fd46427dcc2b25314ad97395bd04287965847af6166d7 not found: ID does not exist"
Sep 08 13:40:40.932001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.931942    2583 scope.go:117] "RemoveContainer" containerID="6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb"
Sep 08 13:40:40.932235 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:40:40.932217    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb\": container with ID starting with 6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb not found: ID does not exist" containerID="6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb"
Sep 08 13:40:40.932300 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.932243    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb"} err="failed to get container status \"6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb\": rpc error: code = NotFound desc = could not find container \"6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb\": container with ID starting with 6d2c74a5d10d776e30bef14b4cfed1e21d79bd350332f8fc10e4596df117fdfb not found: ID does not exist"
Sep 08 13:40:40.932300 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.932263    2583 scope.go:117] "RemoveContainer" containerID="69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4"
Sep 08 13:40:40.932599 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:40:40.932562    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4\": container with ID starting with 69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4 not found: ID does not exist" containerID="69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4"
Sep 08 13:40:40.932599 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.932590    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4"} err="failed to get container status \"69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4\": rpc error: code = NotFound desc = could not find container \"69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4\": container with ID starting with 69d14ed07b18e86cd70f9fe45a1769341f08913d031676a3030c5cf25a5c07b4 not found: ID does not exist"
Sep 08 13:40:40.935437 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.935413    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"]
Sep 08 13:40:40.940849 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:40.940828    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-4btsx"]
Sep 08 13:40:41.426321 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:41.426290    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:41.594914 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:41.594874    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" path="/var/lib/kubelet/pods/fcbefcaa-b854-4afd-9afb-9662e5c9dda5/volumes"
Sep 08 13:40:42.107398 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:42.107315    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:40:57.670331 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.670299    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-03ad552f"]
Sep 08 13:40:57.670814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.670657    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kube-rbac-proxy"
Sep 08 13:40:57.670814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.670671    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kube-rbac-proxy"
Sep 08 13:40:57.670814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.670691    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="storage-initializer"
Sep 08 13:40:57.670814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.670696    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="storage-initializer"
Sep 08 13:40:57.670814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.670703    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kserve-container"
Sep 08 13:40:57.670814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.670709    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kserve-container"
Sep 08 13:40:57.670814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.670769    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kserve-container"
Sep 08 13:40:57.670814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.670778    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcbefcaa-b854-4afd-9afb-9662e5c9dda5" containerName="kube-rbac-proxy"
Sep 08 13:40:57.675311 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.675294    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-03ad552f"
Sep 08 13:40:57.681567 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.681541    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-03ad552f"]
Sep 08 13:40:57.754250 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.754209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbmzb\" (UniqueName: \"kubernetes.io/projected/e3cc7bd8-0051-4890-99b9-9cb295b6c323-kube-api-access-kbmzb\") pod \"curl-metrics-03ad552f\" (UID: \"e3cc7bd8-0051-4890-99b9-9cb295b6c323\") " pod="kserve/curl-metrics-03ad552f"
Sep 08 13:40:57.855669 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.855623    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbmzb\" (UniqueName: \"kubernetes.io/projected/e3cc7bd8-0051-4890-99b9-9cb295b6c323-kube-api-access-kbmzb\") pod \"curl-metrics-03ad552f\" (UID: \"e3cc7bd8-0051-4890-99b9-9cb295b6c323\") " pod="kserve/curl-metrics-03ad552f"
Sep 08 13:40:57.864721 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.864695    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbmzb\" (UniqueName: \"kubernetes.io/projected/e3cc7bd8-0051-4890-99b9-9cb295b6c323-kube-api-access-kbmzb\") pod \"curl-metrics-03ad552f\" (UID: \"e3cc7bd8-0051-4890-99b9-9cb295b6c323\") " pod="kserve/curl-metrics-03ad552f"
Sep 08 13:40:57.992236 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:57.992155    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-03ad552f"
Sep 08 13:40:58.266888 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:58.266863    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-03ad552f"]
Sep 08 13:40:58.269605 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:40:58.269575    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3cc7bd8_0051_4890_99b9_9cb295b6c323.slice/crio-a7942d4cb237d8411332f750fcf473ae1e0a4332f68d1ff74a923737ed8b30e1 WatchSource:0}: Error finding container a7942d4cb237d8411332f750fcf473ae1e0a4332f68d1ff74a923737ed8b30e1: Status 404 returned error can't find the container with id a7942d4cb237d8411332f750fcf473ae1e0a4332f68d1ff74a923737ed8b30e1
Sep 08 13:40:58.272043 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:58.272023    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:40:58.986463 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:58.986428    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-03ad552f" event={"ID":"e3cc7bd8-0051-4890-99b9-9cb295b6c323","Type":"ContainerStarted","Data":"a7942d4cb237d8411332f750fcf473ae1e0a4332f68d1ff74a923737ed8b30e1"}
Sep 08 13:40:59.993025 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:59.992982    2583 generic.go:356] "Generic (PLEG): container finished" podID="e3cc7bd8-0051-4890-99b9-9cb295b6c323" containerID="228ff388c79776a7f1f03454fedd3139865eecf7712f3e14ddf34c0246df60ea" exitCode=0
Sep 08 13:40:59.993412 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:40:59.993074    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-03ad552f" event={"ID":"e3cc7bd8-0051-4890-99b9-9cb295b6c323","Type":"ContainerDied","Data":"228ff388c79776a7f1f03454fedd3139865eecf7712f3e14ddf34c0246df60ea"}
Sep 08 13:41:01.123374 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:01.123351    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-03ad552f"
Sep 08 13:41:01.146774 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:01.146737    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-03ad552f_e3cc7bd8-0051-4890-99b9-9cb295b6c323/curl-metrics-03ad552f/0.log"
Sep 08 13:41:01.171348 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:01.171314    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-03ad552f"]
Sep 08 13:41:01.174936 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:01.174905    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-03ad552f"]
Sep 08 13:41:01.286423 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:01.286335    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbmzb\" (UniqueName: \"kubernetes.io/projected/e3cc7bd8-0051-4890-99b9-9cb295b6c323-kube-api-access-kbmzb\") pod \"e3cc7bd8-0051-4890-99b9-9cb295b6c323\" (UID: \"e3cc7bd8-0051-4890-99b9-9cb295b6c323\") "
Sep 08 13:41:01.288607 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:01.288582    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3cc7bd8-0051-4890-99b9-9cb295b6c323-kube-api-access-kbmzb" (OuterVolumeSpecName: "kube-api-access-kbmzb") pod "e3cc7bd8-0051-4890-99b9-9cb295b6c323" (UID: "e3cc7bd8-0051-4890-99b9-9cb295b6c323"). InnerVolumeSpecName "kube-api-access-kbmzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:41:01.387697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:01.387663    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbmzb\" (UniqueName: \"kubernetes.io/projected/e3cc7bd8-0051-4890-99b9-9cb295b6c323-kube-api-access-kbmzb\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:41:01.595144 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:01.595110    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3cc7bd8-0051-4890-99b9-9cb295b6c323" path="/var/lib/kubelet/pods/e3cc7bd8-0051-4890-99b9-9cb295b6c323/volumes"
Sep 08 13:41:02.003900 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:02.003815    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-03ad552f"
Sep 08 13:41:02.003900 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:02.003825    2583 scope.go:117] "RemoveContainer" containerID="228ff388c79776a7f1f03454fedd3139865eecf7712f3e14ddf34c0246df60ea"
Sep 08 13:41:06.581395 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.581354    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-01f86cde"]
Sep 08 13:41:06.581832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.581722    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3cc7bd8-0051-4890-99b9-9cb295b6c323" containerName="curl-metrics-03ad552f"
Sep 08 13:41:06.581832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.581736    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3cc7bd8-0051-4890-99b9-9cb295b6c323" containerName="curl-metrics-03ad552f"
Sep 08 13:41:06.581832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.581800    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3cc7bd8-0051-4890-99b9-9cb295b6c323" containerName="curl-metrics-03ad552f"
Sep 08 13:41:06.585960 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.585939    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-01f86cde"
Sep 08 13:41:06.592148 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.592117    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-01f86cde"]
Sep 08 13:41:06.738433 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.738396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mh7\" (UniqueName: \"kubernetes.io/projected/2b802cf9-7910-44a5-a984-15c5d57b9ebf-kube-api-access-57mh7\") pod \"curl-metrics-01f86cde\" (UID: \"2b802cf9-7910-44a5-a984-15c5d57b9ebf\") " pod="kserve/curl-metrics-01f86cde"
Sep 08 13:41:06.839481 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.839384    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mh7\" (UniqueName: \"kubernetes.io/projected/2b802cf9-7910-44a5-a984-15c5d57b9ebf-kube-api-access-57mh7\") pod \"curl-metrics-01f86cde\" (UID: \"2b802cf9-7910-44a5-a984-15c5d57b9ebf\") " pod="kserve/curl-metrics-01f86cde"
Sep 08 13:41:06.849045 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.849002    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mh7\" (UniqueName: \"kubernetes.io/projected/2b802cf9-7910-44a5-a984-15c5d57b9ebf-kube-api-access-57mh7\") pod \"curl-metrics-01f86cde\" (UID: \"2b802cf9-7910-44a5-a984-15c5d57b9ebf\") " pod="kserve/curl-metrics-01f86cde"
Sep 08 13:41:06.903380 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:06.903341    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-01f86cde"
Sep 08 13:41:07.175376 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:07.175281    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-01f86cde"]
Sep 08 13:41:07.178037 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:41:07.177991    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b802cf9_7910_44a5_a984_15c5d57b9ebf.slice/crio-bad4a7e33141ed6580d409eed3ea006f997ea986a06860c94d0f0ebbbc6dd5bc WatchSource:0}: Error finding container bad4a7e33141ed6580d409eed3ea006f997ea986a06860c94d0f0ebbbc6dd5bc: Status 404 returned error can't find the container with id bad4a7e33141ed6580d409eed3ea006f997ea986a06860c94d0f0ebbbc6dd5bc
Sep 08 13:41:08.032862 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:08.032830    2583 generic.go:356] "Generic (PLEG): container finished" podID="2b802cf9-7910-44a5-a984-15c5d57b9ebf" containerID="6e2a7ae29441870fc7bacee9bee4540650fdf6ed952734b18aeb104e0d7ee213" exitCode=0
Sep 08 13:41:08.033238 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:08.032926    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-01f86cde" event={"ID":"2b802cf9-7910-44a5-a984-15c5d57b9ebf","Type":"ContainerDied","Data":"6e2a7ae29441870fc7bacee9bee4540650fdf6ed952734b18aeb104e0d7ee213"}
Sep 08 13:41:08.033238 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:08.032974    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-01f86cde" event={"ID":"2b802cf9-7910-44a5-a984-15c5d57b9ebf","Type":"ContainerStarted","Data":"bad4a7e33141ed6580d409eed3ea006f997ea986a06860c94d0f0ebbbc6dd5bc"}
Sep 08 13:41:09.164979 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:09.164955    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-01f86cde"
Sep 08 13:41:09.188624 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:09.188599    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-01f86cde_2b802cf9-7910-44a5-a984-15c5d57b9ebf/curl-metrics-01f86cde/0.log"
Sep 08 13:41:09.211222 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:09.211192    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-01f86cde"]
Sep 08 13:41:09.214975 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:09.214952    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-01f86cde"]
Sep 08 13:41:09.261166 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:09.261135    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57mh7\" (UniqueName: \"kubernetes.io/projected/2b802cf9-7910-44a5-a984-15c5d57b9ebf-kube-api-access-57mh7\") pod \"2b802cf9-7910-44a5-a984-15c5d57b9ebf\" (UID: \"2b802cf9-7910-44a5-a984-15c5d57b9ebf\") "
Sep 08 13:41:09.263389 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:09.263360    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b802cf9-7910-44a5-a984-15c5d57b9ebf-kube-api-access-57mh7" (OuterVolumeSpecName: "kube-api-access-57mh7") pod "2b802cf9-7910-44a5-a984-15c5d57b9ebf" (UID: "2b802cf9-7910-44a5-a984-15c5d57b9ebf"). InnerVolumeSpecName "kube-api-access-57mh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:41:09.362642 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:09.362609    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57mh7\" (UniqueName: \"kubernetes.io/projected/2b802cf9-7910-44a5-a984-15c5d57b9ebf-kube-api-access-57mh7\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:41:09.595467 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:09.595422    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b802cf9-7910-44a5-a984-15c5d57b9ebf" path="/var/lib/kubelet/pods/2b802cf9-7910-44a5-a984-15c5d57b9ebf/volumes"
Sep 08 13:41:10.042406 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:10.042378    2583 scope.go:117] "RemoveContainer" containerID="6e2a7ae29441870fc7bacee9bee4540650fdf6ed952734b18aeb104e0d7ee213"
Sep 08 13:41:10.042594 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:10.042388    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-01f86cde"
Sep 08 13:41:11.625936 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:11.625911    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:41:11.627247 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:11.627228    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:41:14.581142 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:14.581092    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:41:15.476580 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:41:15.476544    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:13.098032 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:13.097976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:18.314177 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:18.314143    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:18.468136 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:18.468086    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:21.193493 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:21.193452    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:35.845833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:35.845791    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:44.742503 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:44.742462    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:49.976490 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:49.976440    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:50.028694 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.028652    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"]
Sep 08 13:42:50.029096 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.029079    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b802cf9-7910-44a5-a984-15c5d57b9ebf" containerName="curl-metrics-01f86cde"
Sep 08 13:42:50.029232 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.029098    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b802cf9-7910-44a5-a984-15c5d57b9ebf" containerName="curl-metrics-01f86cde"
Sep 08 13:42:50.029232 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.029219    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b802cf9-7910-44a5-a984-15c5d57b9ebf" containerName="curl-metrics-01f86cde"
Sep 08 13:42:50.032393 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.032369    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.035493 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.035465    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:42:50.035633 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.035527    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 08 13:42:50.035633 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.035559    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:42:50.036074 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.036057    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:42:50.036354 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.036339    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 08 13:42:50.036596 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.036579    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:42:50.036682 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.036667    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:42:50.050593 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.050561    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"]
Sep 08 13:42:50.113545 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.113506    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.113750 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.113574    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca101c8-9f3f-4e30-ad44-24f9530a14aa-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.113750 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.113670    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.113750 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.113722    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.113884 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.113790    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xhgh\" (UniqueName: \"kubernetes.io/projected/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-api-access-2xhgh\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.214420 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.214375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xhgh\" (UniqueName: \"kubernetes.io/projected/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-api-access-2xhgh\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.214420 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.214425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.214688 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.214471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca101c8-9f3f-4e30-ad44-24f9530a14aa-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.214688 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.214503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.214688 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.214534    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.214986 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.214963    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.215320 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.215302    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.215374 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.215304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.217216 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.217197    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca101c8-9f3f-4e30-ad44-24f9530a14aa-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.219613 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.219586    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:42:50.225221 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.225193    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xhgh\" (UniqueName: \"kubernetes.io/projected/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-api-access-2xhgh\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.343439 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.343402    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:42:50.628281 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:50.628202    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"]
Sep 08 13:42:50.631736 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:42:50.631709    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca101c8_9f3f_4e30_ad44_24f9530a14aa.slice/crio-f6ab06298ef65a2efe988959b57a51f6b0a6fc8783b87dd5a2228729de6293ee WatchSource:0}: Error finding container f6ab06298ef65a2efe988959b57a51f6b0a6fc8783b87dd5a2228729de6293ee: Status 404 returned error can't find the container with id f6ab06298ef65a2efe988959b57a51f6b0a6fc8783b87dd5a2228729de6293ee
Sep 08 13:42:51.504992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:51.504956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" event={"ID":"dca101c8-9f3f-4e30-ad44-24f9530a14aa","Type":"ContainerStarted","Data":"024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f"}
Sep 08 13:42:51.505414 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:51.505032    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" event={"ID":"dca101c8-9f3f-4e30-ad44-24f9530a14aa","Type":"ContainerStarted","Data":"f6ab06298ef65a2efe988959b57a51f6b0a6fc8783b87dd5a2228729de6293ee"}
Sep 08 13:42:55.524767 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:55.524732    2583 generic.go:356] "Generic (PLEG): container finished" podID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerID="024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f" exitCode=0
Sep 08 13:42:55.525175 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:42:55.524779    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" event={"ID":"dca101c8-9f3f-4e30-ad44-24f9530a14aa","Type":"ContainerDied","Data":"024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f"}
Sep 08 13:44:14.432863 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:44:14.432813    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:19.583132 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:44:19.583092    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:19.688543 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:44:19.688506    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:44:57.188219 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:44:57.188180    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" event={"ID":"dca101c8-9f3f-4e30-ad44-24f9530a14aa","Type":"ContainerStarted","Data":"77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3"}
Sep 08 13:44:57.188694 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:44:57.188253    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" event={"ID":"dca101c8-9f3f-4e30-ad44-24f9530a14aa","Type":"ContainerStarted","Data":"6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44"}
Sep 08 13:44:57.188694 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:44:57.188294    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:44:57.221305 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:44:57.221230    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" podStartSLOduration=3.695262783 podStartE2EDuration="2m8.22121057s" totalImagesPullingTime="2m0.673680189s" totalInitContainerRuntime="3.852267598s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:42:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:44:57.220812546 +0000 UTC m=+1726.149716315" watchObservedRunningTime="2026-09-08 13:44:57.22121057 +0000 UTC m=+1726.150114313"
Sep 08 13:44:58.193025 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:44:58.192993    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:45:01.863566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:01.863530    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:04.205724 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:04.205695    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:45:07.265951 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:07.265902    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:07.531035 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:07.530928    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:27.823685 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:27.823643    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:34.212362 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:34.212314    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 08 13:45:42.253706 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:42.253672    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:44.209026 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:44.208985    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:45:47.561337 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.561302    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:47.804296 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.804253    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:47.853626 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.853594    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"]
Sep 08 13:45:47.856888 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.856856    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:47.859814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.859786    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 08 13:45:47.860121 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.860104    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 08 13:45:47.877210 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.877184    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"]
Sep 08 13:45:47.989788 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.989746    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:47.990005 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.989821    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:47.990005 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.989845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v784n\" (UniqueName: \"kubernetes.io/projected/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-api-access-v784n\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:47.990005 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.989869    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:47.990005 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:47.989967    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.090699 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.090648    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.090699 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.090699    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v784n\" (UniqueName: \"kubernetes.io/projected/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-api-access-v784n\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.090974 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.090727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.090974 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.090768    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.090974 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.090800    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.090974 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:45:48.090952    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 08 13:45:48.091238 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:45:48.091058    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-proxy-tls podName:3ec88006-bc66-4987-bdcd-6b6c1b13d2e5 nodeName:}" failed. No retries permitted until 2026-09-08 13:45:48.591037652 +0000 UTC m=+1777.519941374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-proxy-tls") pod "isvc-canary-create-predictor-8488566dd5-w4nnt" (UID: "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 08 13:45:48.091445 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.091424    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kserve-provision-location\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.091541 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.091520    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-cabundle-cert\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.091599 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.091573    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.104827 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.104757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v784n\" (UniqueName: \"kubernetes.io/projected/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-api-access-v784n\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.156210 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.156168    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:48.254667 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.254620    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:48.595638 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.595601    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.598264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.598245    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-proxy-tls\") pod \"isvc-canary-create-predictor-8488566dd5-w4nnt\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:48.768159 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:48.768131    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:45:49.059228 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:49.059201    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"]
Sep 08 13:45:49.061525 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:45:49.061497    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec88006_bc66_4987_bdcd_6b6c1b13d2e5.slice/crio-40b52a60e9298e29719c0ee93fb2ca3968757cf700dd9545ac058b3c146128c5 WatchSource:0}: Error finding container 40b52a60e9298e29719c0ee93fb2ca3968757cf700dd9545ac058b3c146128c5: Status 404 returned error can't find the container with id 40b52a60e9298e29719c0ee93fb2ca3968757cf700dd9545ac058b3c146128c5
Sep 08 13:45:49.435242 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:49.435209    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" event={"ID":"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5","Type":"ContainerStarted","Data":"e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012"}
Sep 08 13:45:49.435431 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:49.435265    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" event={"ID":"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5","Type":"ContainerStarted","Data":"40b52a60e9298e29719c0ee93fb2ca3968757cf700dd9545ac058b3c146128c5"}
Sep 08 13:45:50.176625 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:50.176592    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"]
Sep 08 13:45:50.177042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:50.176846    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kserve-container" containerID="cri-o://6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44" gracePeriod=30
Sep 08 13:45:50.177042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:50.176871    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kube-rbac-proxy" containerID="cri-o://77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3" gracePeriod=30
Sep 08 13:45:50.188074 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:50.188047    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:50.442383 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:50.442300    2583 generic.go:356] "Generic (PLEG): container finished" podID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerID="77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3" exitCode=2
Sep 08 13:45:50.442537 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:50.442375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" event={"ID":"dca101c8-9f3f-4e30-ad44-24f9530a14aa","Type":"ContainerDied","Data":"77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3"}
Sep 08 13:45:53.457566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:53.457528    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerID="e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012" exitCode=0
Sep 08 13:45:53.457955 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:53.457591    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" event={"ID":"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5","Type":"ContainerDied","Data":"e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012"}
Sep 08 13:45:54.200792 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:54.200745    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 08 13:45:54.206510 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:54.206470    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.34:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.132.0.34:8080: connect: connection refused"
Sep 08 13:45:55.657997 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:55.657958    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:55.760621 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:55.760496    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:45:59.200959 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.200915    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused"
Sep 08 13:45:59.370474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.370443    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:45:59.398953 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.398918    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-rbac-proxy-sar-config\") pod \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") "
Sep 08 13:45:59.399176 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.398982    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-cabundle-cert\") pod \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") "
Sep 08 13:45:59.399176 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.399046    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca101c8-9f3f-4e30-ad44-24f9530a14aa-proxy-tls\") pod \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") "
Sep 08 13:45:59.399176 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.399144    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kserve-provision-location\") pod \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") "
Sep 08 13:45:59.399176 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.399169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xhgh\" (UniqueName: \"kubernetes.io/projected/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-api-access-2xhgh\") pod \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\" (UID: \"dca101c8-9f3f-4e30-ad44-24f9530a14aa\") "
Sep 08 13:45:59.399504 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.399439    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dca101c8-9f3f-4e30-ad44-24f9530a14aa" (UID: "dca101c8-9f3f-4e30-ad44-24f9530a14aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:59.399504 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.399451    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dca101c8-9f3f-4e30-ad44-24f9530a14aa" (UID: "dca101c8-9f3f-4e30-ad44-24f9530a14aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:45:59.399765 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.399674    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dca101c8-9f3f-4e30-ad44-24f9530a14aa" (UID: "dca101c8-9f3f-4e30-ad44-24f9530a14aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:45:59.402408 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.402372    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca101c8-9f3f-4e30-ad44-24f9530a14aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dca101c8-9f3f-4e30-ad44-24f9530a14aa" (UID: "dca101c8-9f3f-4e30-ad44-24f9530a14aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:45:59.402408 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.402394    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-api-access-2xhgh" (OuterVolumeSpecName: "kube-api-access-2xhgh") pod "dca101c8-9f3f-4e30-ad44-24f9530a14aa" (UID: "dca101c8-9f3f-4e30-ad44-24f9530a14aa"). InnerVolumeSpecName "kube-api-access-2xhgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:45:59.499109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499006    2583 generic.go:356] "Generic (PLEG): container finished" podID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerID="6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44" exitCode=0
Sep 08 13:45:59.499275 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499105    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"
Sep 08 13:45:59.499275 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499121    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" event={"ID":"dca101c8-9f3f-4e30-ad44-24f9530a14aa","Type":"ContainerDied","Data":"6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44"}
Sep 08 13:45:59.499275 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499199    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs" event={"ID":"dca101c8-9f3f-4e30-ad44-24f9530a14aa","Type":"ContainerDied","Data":"f6ab06298ef65a2efe988959b57a51f6b0a6fc8783b87dd5a2228729de6293ee"}
Sep 08 13:45:59.499275 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499235    2583 scope.go:117] "RemoveContainer" containerID="77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3"
Sep 08 13:45:59.499927 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499905    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:45:59.500063 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499930    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xhgh\" (UniqueName: \"kubernetes.io/projected/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-api-access-2xhgh\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:45:59.500063 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499947    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:45:59.500063 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499960    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca101c8-9f3f-4e30-ad44-24f9530a14aa-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:45:59.500063 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.499974    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca101c8-9f3f-4e30-ad44-24f9530a14aa-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:45:59.510087 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.510066    2583 scope.go:117] "RemoveContainer" containerID="6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44"
Sep 08 13:45:59.520162 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.520141    2583 scope.go:117] "RemoveContainer" containerID="024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f"
Sep 08 13:45:59.523853 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.523824    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"]
Sep 08 13:45:59.528426 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.528396    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-bcbgs"]
Sep 08 13:45:59.529171 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.529151    2583 scope.go:117] "RemoveContainer" containerID="77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3"
Sep 08 13:45:59.529499 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:45:59.529477    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3\": container with ID starting with 77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3 not found: ID does not exist" containerID="77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3"
Sep 08 13:45:59.529572 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.529507    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3"} err="failed to get container status \"77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3\": rpc error: code = NotFound desc = could not find container \"77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3\": container with ID starting with 77396f86a3520e4868c6236e2d697140d3d220aceec2abb9d0f6b0196acdfdc3 not found: ID does not exist"
Sep 08 13:45:59.529572 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.529525    2583 scope.go:117] "RemoveContainer" containerID="6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44"
Sep 08 13:45:59.529801 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:45:59.529781    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44\": container with ID starting with 6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44 not found: ID does not exist" containerID="6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44"
Sep 08 13:45:59.529865 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.529812    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44"} err="failed to get container status \"6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44\": rpc error: code = NotFound desc = could not find container \"6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44\": container with ID starting with 6ee93628fc11f34595d6817f17b304a5808cbaddd0b7155b7b98785aa531ed44 not found: ID does not exist"
Sep 08 13:45:59.529865 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.529837    2583 scope.go:117] "RemoveContainer" containerID="024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f"
Sep 08 13:45:59.530196 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:45:59.530175    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f\": container with ID starting with 024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f not found: ID does not exist" containerID="024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f"
Sep 08 13:45:59.530262 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.530199    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f"} err="failed to get container status \"024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f\": rpc error: code = NotFound desc = could not find container \"024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f\": container with ID starting with 024aa70ecc313ec4de9cab4ec1dc50563233b2f04b00d0eebdb60af2bf4ebb8f not found: ID does not exist"
Sep 08 13:45:59.595799 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:45:59.595762    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" path="/var/lib/kubelet/pods/dca101c8-9f3f-4e30-ad44-24f9530a14aa/volumes"
Sep 08 13:46:08.549344 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:08.549305    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" event={"ID":"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5","Type":"ContainerStarted","Data":"91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495"}
Sep 08 13:46:08.549736 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:08.549369    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" event={"ID":"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5","Type":"ContainerStarted","Data":"ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd"}
Sep 08 13:46:08.549736 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:08.549633    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:46:08.549851 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:08.549786    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:46:08.551162 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:08.551134    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:46:08.572973 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:08.572917    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podStartSLOduration=3.709040737 podStartE2EDuration="21.572899645s" totalImagesPullingTime="14.081855919s" totalInitContainerRuntime="3.782002989s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:08.570255282 +0000 UTC m=+1797.499159230" watchObservedRunningTime="2026-09-08 13:46:08.572899645 +0000 UTC m=+1797.501803386"
Sep 08 13:46:09.553793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:09.553751    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:46:11.660694 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:11.660658    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:46:11.661747 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:11.661730    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:46:14.558837 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:14.558803    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:46:14.559367 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:14.559340    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:46:24.560056 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:24.559982    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:46:27.246128 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:27.246089    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:32.316854 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.316812    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:32.360071 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.360004    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"]
Sep 08 13:46:32.360537 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.360515    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kserve-container"
Sep 08 13:46:32.360635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.360539    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kserve-container"
Sep 08 13:46:32.360635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.360560    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="storage-initializer"
Sep 08 13:46:32.360635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.360568    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="storage-initializer"
Sep 08 13:46:32.360635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.360589    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kube-rbac-proxy"
Sep 08 13:46:32.360635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.360598    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kube-rbac-proxy"
Sep 08 13:46:32.360901 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.360692    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kube-rbac-proxy"
Sep 08 13:46:32.360901 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.360706    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca101c8-9f3f-4e30-ad44-24f9530a14aa" containerName="kserve-container"
Sep 08 13:46:32.635747 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.635658    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:46:32.640140 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.640100    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"]
Sep 08 13:46:32.640279 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.640227    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.643559 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.643530    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 08 13:46:32.643712 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.643692    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 08 13:46:32.808132 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.808097    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.808132 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.808133    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.808354 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.808189    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpfgn\" (UniqueName: \"kubernetes.io/projected/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-api-access-qpfgn\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.808354 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.808299    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.808441 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.808355    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.909441 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.909350    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.909441 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.909397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.909441 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.909434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.909741 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.909455    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.909741 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.909489    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpfgn\" (UniqueName: \"kubernetes.io/projected/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-api-access-qpfgn\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.909932 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.909908    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.910148 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.910122    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.910148 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.910140    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.912290 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.912270    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.919960 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.919934    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpfgn\" (UniqueName: \"kubernetes.io/projected/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-api-access-qpfgn\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:32.950692 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:32.950659    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:33.264677 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:33.264487    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"]
Sep 08 13:46:33.267682 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:46:33.267653    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc9ca2e_d9b7_466e_84e8_af9f81e776ce.slice/crio-e4260651e7c33c54b251e1fdea3db454614b5ba47712bdebdd54c796ed5ec663 WatchSource:0}: Error finding container e4260651e7c33c54b251e1fdea3db454614b5ba47712bdebdd54c796ed5ec663: Status 404 returned error can't find the container with id e4260651e7c33c54b251e1fdea3db454614b5ba47712bdebdd54c796ed5ec663
Sep 08 13:46:33.659822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:33.659777    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" event={"ID":"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce","Type":"ContainerStarted","Data":"56f6b3ecaf4c033b91e996e64bccb97d573134f66375c4c0682e32c7cd2888c8"}
Sep 08 13:46:33.660252 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:33.659885    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" event={"ID":"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce","Type":"ContainerStarted","Data":"e4260651e7c33c54b251e1fdea3db454614b5ba47712bdebdd54c796ed5ec663"}
Sep 08 13:46:34.559892 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:34.559846    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:46:37.679974 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:37.679937    2583 generic.go:356] "Generic (PLEG): container finished" podID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerID="56f6b3ecaf4c033b91e996e64bccb97d573134f66375c4c0682e32c7cd2888c8" exitCode=0
Sep 08 13:46:37.680512 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:37.680035    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" event={"ID":"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce","Type":"ContainerDied","Data":"56f6b3ecaf4c033b91e996e64bccb97d573134f66375c4c0682e32c7cd2888c8"}
Sep 08 13:46:38.687105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:38.687070    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" event={"ID":"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce","Type":"ContainerStarted","Data":"72ece63df15184febdbf1e4ccb51de7255f41c2a5ef9d27c80471f31f998a4c8"}
Sep 08 13:46:38.687551 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:38.687137    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" event={"ID":"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce","Type":"ContainerStarted","Data":"c00586181ce14e7069a0765a93a195bf4159fe57778289e6dc8e40897231a6ac"}
Sep 08 13:46:38.687551 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:38.687365    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:38.711722 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:38.711655    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" podStartSLOduration=2.852113993 podStartE2EDuration="6.71163142s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.859517427s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:46:38.710414378 +0000 UTC m=+1827.639318133" watchObservedRunningTime="2026-09-08 13:46:38.71163142 +0000 UTC m=+1827.640535163"
Sep 08 13:46:39.692252 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:39.692221    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:44.560181 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:44.560138    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:46:45.701953 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:45.701925    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:46:54.559990 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:46:54.559950    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:47:04.559819 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:04.559780    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:47:10.990080 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:10.990035    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"]
Sep 08 13:47:10.990581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:10.990297    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" containerID="cri-o://ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd" gracePeriod=30
Sep 08 13:47:10.990581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:10.990358    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy" containerID="cri-o://91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495" gracePeriod=30
Sep 08 13:47:10.990581 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:10.990566    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:47:11.851220 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:11.851185    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerID="91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495" exitCode=2
Sep 08 13:47:11.851220 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:11.851227    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" event={"ID":"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5","Type":"ContainerDied","Data":"91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495"}
Sep 08 13:47:14.555068 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:14.554988    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 08 13:47:15.712943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:15.712897    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 08 13:47:17.704612 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:17.704570    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:17.830290 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:17.830253    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:19.554205 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:19.554153    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 08 13:47:20.991378 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:20.991330    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 08 13:47:22.785659 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:22.785623    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:22.902333 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:22.902300    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:22.948035 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:22.947982    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"]
Sep 08 13:47:22.951621 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:22.951597    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:22.954676 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:22.954653    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 08 13:47:22.954940 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:22.954921    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 08 13:47:22.970538 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:22.970507    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"]
Sep 08 13:47:23.032441 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.032412    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:23.066622 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.066584    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.066783 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.066641    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.066783 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.066752    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqnq7\" (UniqueName: \"kubernetes.io/projected/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-api-access-rqnq7\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.066875 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.066792    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.066931 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.066891    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.168221 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.168183    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqnq7\" (UniqueName: \"kubernetes.io/projected/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-api-access-rqnq7\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.168428 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.168235    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.168500 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.168422    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.168500 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.168473    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.168616 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.168512    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.168923 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.168893    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kserve-provision-location\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.169066 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.168936    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-cabundle-cert\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.169191 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.169174    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.171178 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.171157    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-proxy-tls\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.177781 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.177757    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqnq7\" (UniqueName: \"kubernetes.io/projected/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-api-access-rqnq7\") pod \"isvc-canary-promote-predictor-64849fbd89-pvhpf\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.262677 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.262635    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:23.543272 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.543246    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"]
Sep 08 13:47:23.545822 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:47:23.545792    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f9d55d7_a8e3_41a8_a9df_634a1674d13a.slice/crio-4ecec23e20f980b937ec5c51f16372b5f1a4acdaaba60868bbfe6670d9fd7ac2 WatchSource:0}: Error finding container 4ecec23e20f980b937ec5c51f16372b5f1a4acdaaba60868bbfe6670d9fd7ac2: Status 404 returned error can't find the container with id 4ecec23e20f980b937ec5c51f16372b5f1a4acdaaba60868bbfe6670d9fd7ac2
Sep 08 13:47:23.910494 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.910450    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" event={"ID":"2f9d55d7-a8e3-41a8-a9df-634a1674d13a","Type":"ContainerStarted","Data":"961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42"}
Sep 08 13:47:23.911033 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:23.910520    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" event={"ID":"2f9d55d7-a8e3-41a8-a9df-634a1674d13a","Type":"ContainerStarted","Data":"4ecec23e20f980b937ec5c51f16372b5f1a4acdaaba60868bbfe6670d9fd7ac2"}
Sep 08 13:47:24.554215 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:24.554162    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 08 13:47:24.554432 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:24.554338    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:47:25.705001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:25.704970    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:47:27.930232 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:27.930200    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerID="961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42" exitCode=0
Sep 08 13:47:27.930629 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:27.930275    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" event={"ID":"2f9d55d7-a8e3-41a8-a9df-634a1674d13a","Type":"ContainerDied","Data":"961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42"}
Sep 08 13:47:28.937241 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:28.937197    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" event={"ID":"2f9d55d7-a8e3-41a8-a9df-634a1674d13a","Type":"ContainerStarted","Data":"23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd"}
Sep 08 13:47:28.937627 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:28.937265    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" event={"ID":"2f9d55d7-a8e3-41a8-a9df-634a1674d13a","Type":"ContainerStarted","Data":"9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c"}
Sep 08 13:47:28.937627 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:28.937584    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:28.937720 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:28.937634    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:28.939162 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:28.939134    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:47:28.969476 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:28.969409    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podStartSLOduration=3.153000485 podStartE2EDuration="6.969388105s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.81638762s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:47:28.967537651 +0000 UTC m=+1877.896441402" watchObservedRunningTime="2026-09-08 13:47:28.969388105 +0000 UTC m=+1877.898291848"
Sep 08 13:47:29.554845 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:29.554801    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 08 13:47:29.942138 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:29.942092    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:47:30.992245 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:30.992214    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:47:32.434965 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:32.434889    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"]
Sep 08 13:47:32.435390 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:32.435149    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kserve-container" containerID="cri-o://c00586181ce14e7069a0765a93a195bf4159fe57778289e6dc8e40897231a6ac" gracePeriod=30
Sep 08 13:47:32.435390 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:32.435180    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kube-rbac-proxy" containerID="cri-o://72ece63df15184febdbf1e4ccb51de7255f41c2a5ef9d27c80471f31f998a4c8" gracePeriod=30
Sep 08 13:47:32.479342 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:32.479294    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:32.956772 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:32.956739    2583 generic.go:356] "Generic (PLEG): container finished" podID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerID="72ece63df15184febdbf1e4ccb51de7255f41c2a5ef9d27c80471f31f998a4c8" exitCode=2
Sep 08 13:47:32.956949 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:32.956813    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" event={"ID":"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce","Type":"ContainerDied","Data":"72ece63df15184febdbf1e4ccb51de7255f41c2a5ef9d27c80471f31f998a4c8"}
Sep 08 13:47:34.554802 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:34.554760    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 08 13:47:34.947001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:34.946971    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:47:34.947609 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:34.947584    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:47:35.697396 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:35.697351    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 08 13:47:37.699345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:37.699302    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:37.837874 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:37.837838    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:47:39.554206 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:39.554161    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 08 13:47:39.995373 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:39.995339    2583 generic.go:356] "Generic (PLEG): container finished" podID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerID="c00586181ce14e7069a0765a93a195bf4159fe57778289e6dc8e40897231a6ac" exitCode=0
Sep 08 13:47:39.995509 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:39.995411    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" event={"ID":"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce","Type":"ContainerDied","Data":"c00586181ce14e7069a0765a93a195bf4159fe57778289e6dc8e40897231a6ac"}
Sep 08 13:47:40.083655 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.083633    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:47:40.111025 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.110988    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kserve-provision-location\") pod \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") "
Sep 08 13:47:40.111179 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.111047    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-proxy-tls\") pod \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") "
Sep 08 13:47:40.111179 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.111079    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpfgn\" (UniqueName: \"kubernetes.io/projected/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-api-access-qpfgn\") pod \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") "
Sep 08 13:47:40.111179 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.111122    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-cabundle-cert\") pod \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") "
Sep 08 13:47:40.111353 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.111190    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-rbac-proxy-sar-config\") pod \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\" (UID: \"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce\") "
Sep 08 13:47:40.111444 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.111341    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" (UID: "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:40.111547 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.111525    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" (UID: "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:40.111635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.111609    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" (UID: "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:40.113578 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.113552    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-api-access-qpfgn" (OuterVolumeSpecName: "kube-api-access-qpfgn") pod "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" (UID: "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce"). InnerVolumeSpecName "kube-api-access-qpfgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:40.113653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.113593    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" (UID: "2bc9ca2e-d9b7-466e-84e8-af9f81e776ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:40.212793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.212694    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:40.212793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.212728    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:40.212793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.212744    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:40.212793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.212757    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpfgn\" (UniqueName: \"kubernetes.io/projected/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-kube-api-access-qpfgn\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:40.212793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:40.212771    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:41.002381 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.002341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj" event={"ID":"2bc9ca2e-d9b7-466e-84e8-af9f81e776ce","Type":"ContainerDied","Data":"e4260651e7c33c54b251e1fdea3db454614b5ba47712bdebdd54c796ed5ec663"}
Sep 08 13:47:41.002381 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.002370    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"
Sep 08 13:47:41.002808 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.002425    2583 scope.go:117] "RemoveContainer" containerID="72ece63df15184febdbf1e4ccb51de7255f41c2a5ef9d27c80471f31f998a4c8"
Sep 08 13:47:41.011890 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.011872    2583 scope.go:117] "RemoveContainer" containerID="c00586181ce14e7069a0765a93a195bf4159fe57778289e6dc8e40897231a6ac"
Sep 08 13:47:41.019549 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.019532    2583 scope.go:117] "RemoveContainer" containerID="56f6b3ecaf4c033b91e996e64bccb97d573134f66375c4c0682e32c7cd2888c8"
Sep 08 13:47:41.039331 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.039304    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"]
Sep 08 13:47:41.053872 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.053848    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-6b5qj"]
Sep 08 13:47:41.595813 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.595781    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" path="/var/lib/kubelet/pods/2bc9ca2e-d9b7-466e-84e8-af9f81e776ce/volumes"
Sep 08 13:47:41.648780 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.648758    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:47:41.724714 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.724686    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-rbac-proxy-sar-config\") pod \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") "
Sep 08 13:47:41.724981 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.724734    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v784n\" (UniqueName: \"kubernetes.io/projected/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-api-access-v784n\") pod \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") "
Sep 08 13:47:41.724981 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.724771    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-proxy-tls\") pod \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") "
Sep 08 13:47:41.724981 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.724892    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-cabundle-cert\") pod \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") "
Sep 08 13:47:41.724981 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.724940    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kserve-provision-location\") pod \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\" (UID: \"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5\") "
Sep 08 13:47:41.725248 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.725150    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" (UID: "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:41.725308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.725273    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:41.725308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.725269    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" (UID: "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:47:41.725308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.725289    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" (UID: "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:47:41.727455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.727430    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" (UID: "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:47:41.727514 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.727462    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-api-access-v784n" (OuterVolumeSpecName: "kube-api-access-v784n") pod "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" (UID: "3ec88006-bc66-4987-bdcd-6b6c1b13d2e5"). InnerVolumeSpecName "kube-api-access-v784n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:47:41.826651 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.826617    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v784n\" (UniqueName: \"kubernetes.io/projected/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kube-api-access-v784n\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:41.826651 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.826650    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:41.826857 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.826661    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:41.826857 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:41.826669    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:47:42.008346 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.008313    2583 generic.go:356] "Generic (PLEG): container finished" podID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerID="ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd" exitCode=137
Sep 08 13:47:42.008811 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.008378    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"
Sep 08 13:47:42.008811 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.008395    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" event={"ID":"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5","Type":"ContainerDied","Data":"ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd"}
Sep 08 13:47:42.008811 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.008462    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt" event={"ID":"3ec88006-bc66-4987-bdcd-6b6c1b13d2e5","Type":"ContainerDied","Data":"40b52a60e9298e29719c0ee93fb2ca3968757cf700dd9545ac058b3c146128c5"}
Sep 08 13:47:42.008811 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.008504    2583 scope.go:117] "RemoveContainer" containerID="91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495"
Sep 08 13:47:42.017064 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.017047    2583 scope.go:117] "RemoveContainer" containerID="ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd"
Sep 08 13:47:42.024966 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.024946    2583 scope.go:117] "RemoveContainer" containerID="e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012"
Sep 08 13:47:42.032985 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.032960    2583 scope.go:117] "RemoveContainer" containerID="91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495"
Sep 08 13:47:42.033331 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:47:42.033309    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495\": container with ID starting with 91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495 not found: ID does not exist" containerID="91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495"
Sep 08 13:47:42.033409 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.033340    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495"} err="failed to get container status \"91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495\": rpc error: code = NotFound desc = could not find container \"91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495\": container with ID starting with 91475f5b7f065a9abfdc008a38f12879877e0ec86fb97702f200799285fd5495 not found: ID does not exist"
Sep 08 13:47:42.033409 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.033357    2583 scope.go:117] "RemoveContainer" containerID="ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd"
Sep 08 13:47:42.033657 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:47:42.033637    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd\": container with ID starting with ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd not found: ID does not exist" containerID="ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd"
Sep 08 13:47:42.033704 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.033664    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd"} err="failed to get container status \"ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd\": rpc error: code = NotFound desc = could not find container \"ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd\": container with ID starting with ea520d0275da7edc243fe59aa0a30a2f66de3ac5311be652271c812c231c7edd not found: ID does not exist"
Sep 08 13:47:42.033704 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.033680    2583 scope.go:117] "RemoveContainer" containerID="e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012"
Sep 08 13:47:42.033938 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:47:42.033917    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012\": container with ID starting with e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012 not found: ID does not exist" containerID="e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012"
Sep 08 13:47:42.034045 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.033952    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012"} err="failed to get container status \"e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012\": rpc error: code = NotFound desc = could not find container \"e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012\": container with ID starting with e50e02905c6e54e023924303305253777dc99d1404a4e26f46a2caf2238dc012 not found: ID does not exist"
Sep 08 13:47:42.034124 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.034105    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"]
Sep 08 13:47:42.038401 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:42.038378    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-8488566dd5-w4nnt"]
Sep 08 13:47:43.595924 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:43.595885    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" path="/var/lib/kubelet/pods/3ec88006-bc66-4987-bdcd-6b6c1b13d2e5/volumes"
Sep 08 13:47:44.947895 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:44.947854    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:47:54.948165 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:47:54.948107    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:48:04.948059 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:04.947997    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:48:14.948044 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:14.947974    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:48:24.947746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:24.947703    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:48:34.948368 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:34.948322    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:48:38.590023 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:38.589977    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 08 13:48:46.491152 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:46.491108    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"]
Sep 08 13:48:46.491694 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:46.491339    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container" containerID="cri-o://9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c" gracePeriod=30
Sep 08 13:48:46.491694 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:46.491413    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kube-rbac-proxy" containerID="cri-o://23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd" gracePeriod=30
Sep 08 13:48:46.492231 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:46.492211    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:48:47.304052 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:47.303989    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerID="23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd" exitCode=2
Sep 08 13:48:47.304232 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:47.304061    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" event={"ID":"2f9d55d7-a8e3-41a8-a9df-634a1674d13a","Type":"ContainerDied","Data":"23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd"}
Sep 08 13:48:49.942362 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:49.942313    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 08 13:48:51.646689 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.646662    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:48:51.733327 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733231    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-proxy-tls\") pod \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") "
Sep 08 13:48:51.733485 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733333    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kserve-provision-location\") pod \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") "
Sep 08 13:48:51.733485 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733435    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqnq7\" (UniqueName: \"kubernetes.io/projected/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-api-access-rqnq7\") pod \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") "
Sep 08 13:48:51.733485 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733478    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-cabundle-cert\") pod \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") "
Sep 08 13:48:51.733658 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733518    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-rbac-proxy-sar-config\") pod \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\" (UID: \"2f9d55d7-a8e3-41a8-a9df-634a1674d13a\") "
Sep 08 13:48:51.733658 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733609    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f9d55d7-a8e3-41a8-a9df-634a1674d13a" (UID: "2f9d55d7-a8e3-41a8-a9df-634a1674d13a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:48:51.733896 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733846    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f9d55d7-a8e3-41a8-a9df-634a1674d13a" (UID: "2f9d55d7-a8e3-41a8-a9df-634a1674d13a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:48:51.733896 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733884    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f9d55d7-a8e3-41a8-a9df-634a1674d13a" (UID: "2f9d55d7-a8e3-41a8-a9df-634a1674d13a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:48:51.734066 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733954    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.734066 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.733975    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.735708 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.735677    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-api-access-rqnq7" (OuterVolumeSpecName: "kube-api-access-rqnq7") pod "2f9d55d7-a8e3-41a8-a9df-634a1674d13a" (UID: "2f9d55d7-a8e3-41a8-a9df-634a1674d13a"). InnerVolumeSpecName "kube-api-access-rqnq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:48:51.735868 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.735849    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f9d55d7-a8e3-41a8-a9df-634a1674d13a" (UID: "2f9d55d7-a8e3-41a8-a9df-634a1674d13a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:48:51.834778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.834741    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.834778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.834773    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:48:51.834778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:51.834785    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqnq7\" (UniqueName: \"kubernetes.io/projected/2f9d55d7-a8e3-41a8-a9df-634a1674d13a-kube-api-access-rqnq7\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:48:52.329612 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.329572    2583 generic.go:356] "Generic (PLEG): container finished" podID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerID="9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c" exitCode=0
Sep 08 13:48:52.329822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.329642    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" event={"ID":"2f9d55d7-a8e3-41a8-a9df-634a1674d13a","Type":"ContainerDied","Data":"9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c"}
Sep 08 13:48:52.329822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.329660    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"
Sep 08 13:48:52.329822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.329717    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf" event={"ID":"2f9d55d7-a8e3-41a8-a9df-634a1674d13a","Type":"ContainerDied","Data":"4ecec23e20f980b937ec5c51f16372b5f1a4acdaaba60868bbfe6670d9fd7ac2"}
Sep 08 13:48:52.329822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.329743    2583 scope.go:117] "RemoveContainer" containerID="23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd"
Sep 08 13:48:52.338941 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.338921    2583 scope.go:117] "RemoveContainer" containerID="9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c"
Sep 08 13:48:52.347194 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.347176    2583 scope.go:117] "RemoveContainer" containerID="961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42"
Sep 08 13:48:52.352943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.352916    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"]
Sep 08 13:48:52.356291 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.356256    2583 scope.go:117] "RemoveContainer" containerID="23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd"
Sep 08 13:48:52.356621 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:48:52.356597    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd\": container with ID starting with 23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd not found: ID does not exist" containerID="23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd"
Sep 08 13:48:52.356702 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.356628    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd"} err="failed to get container status \"23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd\": rpc error: code = NotFound desc = could not find container \"23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd\": container with ID starting with 23c353c2b5d12f5963dfa6fc2d47159609779113251206eb15e13ff354158ecd not found: ID does not exist"
Sep 08 13:48:52.356702 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.356646    2583 scope.go:117] "RemoveContainer" containerID="9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c"
Sep 08 13:48:52.356905 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:48:52.356891    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c\": container with ID starting with 9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c not found: ID does not exist" containerID="9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c"
Sep 08 13:48:52.356951 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.356907    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c"} err="failed to get container status \"9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c\": rpc error: code = NotFound desc = could not find container \"9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c\": container with ID starting with 9d1185675cec6c56736aa972d8f6466cf1fd648ff585a1e894fb22f96878a84c not found: ID does not exist"
Sep 08 13:48:52.356951 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.356921    2583 scope.go:117] "RemoveContainer" containerID="961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42"
Sep 08 13:48:52.357236 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:48:52.357216    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42\": container with ID starting with 961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42 not found: ID does not exist" containerID="961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42"
Sep 08 13:48:52.357303 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.357245    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42"} err="failed to get container status \"961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42\": rpc error: code = NotFound desc = could not find container \"961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42\": container with ID starting with 961edf53333b4403fb62c8038353aee4f7ff17473f97257d93081fb22c2bdb42 not found: ID does not exist"
Sep 08 13:48:52.357437 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.357419    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-64849fbd89-pvhpf"]
Sep 08 13:48:52.859788 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.859752    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:48:52.954104 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954066    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"]
Sep 08 13:48:52.954415 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954402    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kube-rbac-proxy"
Sep 08 13:48:52.954458 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954417    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kube-rbac-proxy"
Sep 08 13:48:52.954458 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954435    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container"
Sep 08 13:48:52.954458 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954440    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container"
Sep 08 13:48:52.954458 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954447    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="storage-initializer"
Sep 08 13:48:52.954458 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954453    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="storage-initializer"
Sep 08 13:48:52.954458 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954459    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kserve-container"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954465    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kserve-container"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954475    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954480    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954488    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="storage-initializer"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954493    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="storage-initializer"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954501    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kube-rbac-proxy"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954506    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kube-rbac-proxy"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954516    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954522    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954531    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="storage-initializer"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954536    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="storage-initializer"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954583    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kserve-container"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954592    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kserve-container"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954598    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec88006-bc66-4987-bdcd-6b6c1b13d2e5" containerName="kube-rbac-proxy"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954604    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kube-rbac-proxy"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954612    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bc9ca2e-d9b7-466e-84e8-af9f81e776ce" containerName="kube-rbac-proxy"
Sep 08 13:48:52.954644 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.954618    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" containerName="kserve-container"
Sep 08 13:48:52.959281 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.959258    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:52.961559 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.961533    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:48:52.961697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.961602    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 08 13:48:52.961697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.961628    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:48:52.961697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.961610    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:48:52.961858 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.961762    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:48:52.962044 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.962028    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 08 13:48:52.962130 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.962107    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:48:52.970277 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:52.970258    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"]
Sep 08 13:48:53.045729 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.045693    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.045912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.045769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5291137-c373-4f13-9b85-22781a69496b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.045912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.045799    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.045912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.045830    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8qv\" (UniqueName: \"kubernetes.io/projected/c5291137-c373-4f13-9b85-22781a69496b-kube-api-access-hs8qv\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.045912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.045858    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5291137-c373-4f13-9b85-22781a69496b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.147267 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.147157    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.147267 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.147243    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5291137-c373-4f13-9b85-22781a69496b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.147267 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.147274    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.147580 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.147301    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8qv\" (UniqueName: \"kubernetes.io/projected/c5291137-c373-4f13-9b85-22781a69496b-kube-api-access-hs8qv\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.147580 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.147413    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5291137-c373-4f13-9b85-22781a69496b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.147758 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.147732    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5291137-c373-4f13-9b85-22781a69496b-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.147930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.147911    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.147999 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.147962    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.149923 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.149903    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5291137-c373-4f13-9b85-22781a69496b-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.156285 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.156263    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8qv\" (UniqueName: \"kubernetes.io/projected/c5291137-c373-4f13-9b85-22781a69496b-kube-api-access-hs8qv\") pod \"isvc-canary-promote-v2-predictor-67454bd898-4gsf5\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.270403 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.270368    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:53.545348 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.545320    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"]
Sep 08 13:48:53.547960 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:48:53.547933    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5291137_c373_4f13_9b85_22781a69496b.slice/crio-00fd68c2fe5dcf97bfdc7e56a7fe336b3dbac6dfc61de1b666498be42a05d775 WatchSource:0}: Error finding container 00fd68c2fe5dcf97bfdc7e56a7fe336b3dbac6dfc61de1b666498be42a05d775: Status 404 returned error can't find the container with id 00fd68c2fe5dcf97bfdc7e56a7fe336b3dbac6dfc61de1b666498be42a05d775
Sep 08 13:48:53.596950 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:53.596919    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f9d55d7-a8e3-41a8-a9df-634a1674d13a" path="/var/lib/kubelet/pods/2f9d55d7-a8e3-41a8-a9df-634a1674d13a/volumes"
Sep 08 13:48:54.343359 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:54.343324    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" event={"ID":"c5291137-c373-4f13-9b85-22781a69496b","Type":"ContainerStarted","Data":"8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21"}
Sep 08 13:48:54.343768 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:54.343383    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" event={"ID":"c5291137-c373-4f13-9b85-22781a69496b","Type":"ContainerStarted","Data":"00fd68c2fe5dcf97bfdc7e56a7fe336b3dbac6dfc61de1b666498be42a05d775"}
Sep 08 13:48:58.362029 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:58.361977    2583 generic.go:356] "Generic (PLEG): container finished" podID="c5291137-c373-4f13-9b85-22781a69496b" containerID="8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21" exitCode=0
Sep 08 13:48:58.362421 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:58.362042    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" event={"ID":"c5291137-c373-4f13-9b85-22781a69496b","Type":"ContainerDied","Data":"8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21"}
Sep 08 13:48:59.368829 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:59.368795    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" event={"ID":"c5291137-c373-4f13-9b85-22781a69496b","Type":"ContainerStarted","Data":"612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706"}
Sep 08 13:48:59.369241 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:59.368858    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" event={"ID":"c5291137-c373-4f13-9b85-22781a69496b","Type":"ContainerStarted","Data":"f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423"}
Sep 08 13:48:59.369241 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:59.369112    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:48:59.392282 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:48:59.392233    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podStartSLOduration=3.5898744540000003 podStartE2EDuration="7.392215992s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.802341538s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:48:59.389863434 +0000 UTC m=+1968.318767188" watchObservedRunningTime="2026-09-08 13:48:59.392215992 +0000 UTC m=+1968.321119734"
Sep 08 13:49:00.373896 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:00.373867    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:49:00.375179 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:00.375151    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:49:01.377877 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:01.377832    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:49:06.382409 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:06.382334    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:49:06.382872 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:06.382828    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:49:07.823554 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:07.823522    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:12.051359 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:12.051262    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:12.955423 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:12.955378    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:13.242282 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:13.242186    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:49:16.383572 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:16.383534    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:49:26.383038 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:26.382966    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:49:36.383102 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:36.383060    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:49:46.383697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:46.383653    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:49:56.382934 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:56.382895    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:49:58.412223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:49:58.412176    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:06.383126 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:06.383081    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:50:12.590211 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:12.590150    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 08 13:50:22.590779 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:22.590745    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:50:23.050493 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:23.050407    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"]
Sep 08 13:50:23.050732 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:23.050704    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container" containerID="cri-o://f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423" gracePeriod=30
Sep 08 13:50:23.050795 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:23.050737    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kube-rbac-proxy" containerID="cri-o://612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706" gracePeriod=30
Sep 08 13:50:23.075156 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:23.075119    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:23.760502 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:23.760472    2583 generic.go:356] "Generic (PLEG): container finished" podID="c5291137-c373-4f13-9b85-22781a69496b" containerID="612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706" exitCode=2
Sep 08 13:50:23.760897 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:23.760539    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" event={"ID":"c5291137-c373-4f13-9b85-22781a69496b","Type":"ContainerDied","Data":"612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706"}
Sep 08 13:50:26.378536 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:26.378491    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 08 13:50:28.104761 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.104737    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:50:28.171462 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.171391    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-cabundle-cert\") pod \"c5291137-c373-4f13-9b85-22781a69496b\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") "
Sep 08 13:50:28.171462 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.171429    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs8qv\" (UniqueName: \"kubernetes.io/projected/c5291137-c373-4f13-9b85-22781a69496b-kube-api-access-hs8qv\") pod \"c5291137-c373-4f13-9b85-22781a69496b\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") "
Sep 08 13:50:28.171659 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.171515    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-kube-rbac-proxy-sar-config\") pod \"c5291137-c373-4f13-9b85-22781a69496b\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") "
Sep 08 13:50:28.171659 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.171552    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5291137-c373-4f13-9b85-22781a69496b-proxy-tls\") pod \"c5291137-c373-4f13-9b85-22781a69496b\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") "
Sep 08 13:50:28.171659 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.171607    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5291137-c373-4f13-9b85-22781a69496b-kserve-provision-location\") pod \"c5291137-c373-4f13-9b85-22781a69496b\" (UID: \"c5291137-c373-4f13-9b85-22781a69496b\") "
Sep 08 13:50:28.171822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.171782    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5291137-c373-4f13-9b85-22781a69496b" (UID: "c5291137-c373-4f13-9b85-22781a69496b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:50:28.171929 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.171897    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.172038 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.171902    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5291137-c373-4f13-9b85-22781a69496b" (UID: "c5291137-c373-4f13-9b85-22781a69496b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:50:28.172038 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.171956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5291137-c373-4f13-9b85-22781a69496b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5291137-c373-4f13-9b85-22781a69496b" (UID: "c5291137-c373-4f13-9b85-22781a69496b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:50:28.173886 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.173863    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5291137-c373-4f13-9b85-22781a69496b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5291137-c373-4f13-9b85-22781a69496b" (UID: "c5291137-c373-4f13-9b85-22781a69496b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:50:28.174174 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.174151    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5291137-c373-4f13-9b85-22781a69496b-kube-api-access-hs8qv" (OuterVolumeSpecName: "kube-api-access-hs8qv") pod "c5291137-c373-4f13-9b85-22781a69496b" (UID: "c5291137-c373-4f13-9b85-22781a69496b"). InnerVolumeSpecName "kube-api-access-hs8qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:50:28.273209 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.273168    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5291137-c373-4f13-9b85-22781a69496b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.273209 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.273207    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5291137-c373-4f13-9b85-22781a69496b-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.273413 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.273224    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5291137-c373-4f13-9b85-22781a69496b-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.273413 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.273237    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs8qv\" (UniqueName: \"kubernetes.io/projected/c5291137-c373-4f13-9b85-22781a69496b-kube-api-access-hs8qv\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:50:28.277332 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.277294    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:28.323889 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.323855    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"]
Sep 08 13:50:28.324209 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.324196    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container"
Sep 08 13:50:28.324264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.324210    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container"
Sep 08 13:50:28.324264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.324234    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kube-rbac-proxy"
Sep 08 13:50:28.324264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.324240    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kube-rbac-proxy"
Sep 08 13:50:28.324264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.324246    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="storage-initializer"
Sep 08 13:50:28.324264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.324252    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="storage-initializer"
Sep 08 13:50:28.324433 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.324298    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kserve-container"
Sep 08 13:50:28.324433 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.324308    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5291137-c373-4f13-9b85-22781a69496b" containerName="kube-rbac-proxy"
Sep 08 13:50:28.327471 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.327450    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.330203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.330178    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 08 13:50:28.331656 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.331633    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 08 13:50:28.341974 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.341950    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"]
Sep 08 13:50:28.374071 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.374036    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.374257 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.374095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45178f55-a6b8-4e0f-b846-1ed044e3f973-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.374257 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.374171    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7czl7\" (UniqueName: \"kubernetes.io/projected/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-api-access-7czl7\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.374257 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.374250    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.374387 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.374303    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45178f55-a6b8-4e0f-b846-1ed044e3f973-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.377342 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.377312    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:28.474830 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.474740    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45178f55-a6b8-4e0f-b846-1ed044e3f973-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.474830 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.474806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.475103 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.474850    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45178f55-a6b8-4e0f-b846-1ed044e3f973-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.475103 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.474876    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7czl7\" (UniqueName: \"kubernetes.io/projected/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-api-access-7czl7\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.475103 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:50:28.474918    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 08 13:50:28.475103 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:50:28.474985    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45178f55-a6b8-4e0f-b846-1ed044e3f973-proxy-tls podName:45178f55-a6b8-4e0f-b846-1ed044e3f973 nodeName:}" failed. No retries permitted until 2026-09-08 13:50:28.974966689 +0000 UTC m=+2057.903870409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/45178f55-a6b8-4e0f-b846-1ed044e3f973-proxy-tls") pod "isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" (UID: "45178f55-a6b8-4e0f-b846-1ed044e3f973") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 08 13:50:28.475103 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.474928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.475366 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.475336    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45178f55-a6b8-4e0f-b846-1ed044e3f973-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.475571 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.475551    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.475702 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.475682    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.484392 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.484364    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7czl7\" (UniqueName: \"kubernetes.io/projected/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-api-access-7czl7\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.490567 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.490520    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:28.602569 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.602534    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:28.789692 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.789599    2583 generic.go:356] "Generic (PLEG): container finished" podID="c5291137-c373-4f13-9b85-22781a69496b" containerID="f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423" exitCode=0
Sep 08 13:50:28.789692 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.789670    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"
Sep 08 13:50:28.789692 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.789676    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" event={"ID":"c5291137-c373-4f13-9b85-22781a69496b","Type":"ContainerDied","Data":"f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423"}
Sep 08 13:50:28.789964 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.789753    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5" event={"ID":"c5291137-c373-4f13-9b85-22781a69496b","Type":"ContainerDied","Data":"00fd68c2fe5dcf97bfdc7e56a7fe336b3dbac6dfc61de1b666498be42a05d775"}
Sep 08 13:50:28.789964 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.789780    2583 scope.go:117] "RemoveContainer" containerID="612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706"
Sep 08 13:50:28.798693 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.798675    2583 scope.go:117] "RemoveContainer" containerID="f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423"
Sep 08 13:50:28.806454 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.806436    2583 scope.go:117] "RemoveContainer" containerID="8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21"
Sep 08 13:50:28.812429 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.812403    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"]
Sep 08 13:50:28.814739 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.814725    2583 scope.go:117] "RemoveContainer" containerID="612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706"
Sep 08 13:50:28.815413 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:50:28.815285    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706\": container with ID starting with 612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706 not found: ID does not exist" containerID="612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706"
Sep 08 13:50:28.815413 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.815319    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706"} err="failed to get container status \"612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706\": rpc error: code = NotFound desc = could not find container \"612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706\": container with ID starting with 612882ee59df06afc10432a91771e469067ff41a5a0495e2f11044e3f7b7d706 not found: ID does not exist"
Sep 08 13:50:28.815413 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.815343    2583 scope.go:117] "RemoveContainer" containerID="f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423"
Sep 08 13:50:28.815705 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:50:28.815657    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423\": container with ID starting with f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423 not found: ID does not exist" containerID="f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423"
Sep 08 13:50:28.815705 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.815684    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423"} err="failed to get container status \"f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423\": rpc error: code = NotFound desc = could not find container \"f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423\": container with ID starting with f4ecd64b5db5521d3f1594870367620f12140d0bddb88199e6d01ec395b7e423 not found: ID does not exist"
Sep 08 13:50:28.815797 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.815706    2583 scope.go:117] "RemoveContainer" containerID="8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21"
Sep 08 13:50:28.815976 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:50:28.815958    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21\": container with ID starting with 8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21 not found: ID does not exist" containerID="8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21"
Sep 08 13:50:28.816082 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.815983    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21"} err="failed to get container status \"8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21\": rpc error: code = NotFound desc = could not find container \"8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21\": container with ID starting with 8df742b626421b57629c58abe286cfdf51f55bace84e98f5cc3e336246e34a21 not found: ID does not exist"
Sep 08 13:50:28.817109 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.817093    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-67454bd898-4gsf5"]
Sep 08 13:50:28.979614 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.979578    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45178f55-a6b8-4e0f-b846-1ed044e3f973-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:28.982399 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:28.982371    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45178f55-a6b8-4e0f-b846-1ed044e3f973-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-h9mrb\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:29.239753 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:29.239720    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:29.520906 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:29.520772    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"]
Sep 08 13:50:29.523873 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:50:29.523849    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45178f55_a6b8_4e0f_b846_1ed044e3f973.slice/crio-6cddbf4efe3c1e6b2e47d6cdf115836eb5383ff0c7e673ade1355e4d82f04320 WatchSource:0}: Error finding container 6cddbf4efe3c1e6b2e47d6cdf115836eb5383ff0c7e673ade1355e4d82f04320: Status 404 returned error can't find the container with id 6cddbf4efe3c1e6b2e47d6cdf115836eb5383ff0c7e673ade1355e4d82f04320
Sep 08 13:50:29.595818 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:29.595775    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5291137-c373-4f13-9b85-22781a69496b" path="/var/lib/kubelet/pods/c5291137-c373-4f13-9b85-22781a69496b/volumes"
Sep 08 13:50:29.797959 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:29.797857    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" event={"ID":"45178f55-a6b8-4e0f-b846-1ed044e3f973","Type":"ContainerStarted","Data":"1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850"}
Sep 08 13:50:29.797959 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:29.797926    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" event={"ID":"45178f55-a6b8-4e0f-b846-1ed044e3f973","Type":"ContainerStarted","Data":"6cddbf4efe3c1e6b2e47d6cdf115836eb5383ff0c7e673ade1355e4d82f04320"}
Sep 08 13:50:33.815970 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:33.815938    2583 generic.go:356] "Generic (PLEG): container finished" podID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerID="1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850" exitCode=0
Sep 08 13:50:33.816350 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:33.816041    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" event={"ID":"45178f55-a6b8-4e0f-b846-1ed044e3f973","Type":"ContainerDied","Data":"1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850"}
Sep 08 13:50:34.822404 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:34.822356    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" event={"ID":"45178f55-a6b8-4e0f-b846-1ed044e3f973","Type":"ContainerStarted","Data":"2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a"}
Sep 08 13:50:34.822833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:34.822434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" event={"ID":"45178f55-a6b8-4e0f-b846-1ed044e3f973","Type":"ContainerStarted","Data":"799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b"}
Sep 08 13:50:34.822833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:34.822732    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:34.844520 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:34.844474    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podStartSLOduration=3.106700502 podStartE2EDuration="6.844459696s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.737759194s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:34.841998483 +0000 UTC m=+2063.770902224" watchObservedRunningTime="2026-09-08 13:50:34.844459696 +0000 UTC m=+2063.773363437"
Sep 08 13:50:35.827308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:35.827277    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:35.828756 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:35.828726    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 08 13:50:36.831231 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:36.831188    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 08 13:50:41.841487 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:41.837291    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:50:41.841487 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:41.838561    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 08 13:50:43.184207 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:43.184174    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:48.261950 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.261914    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:48.288292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.288259    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"]
Sep 08 13:50:48.291525 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.291502    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.294171 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.294146    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 08 13:50:48.294171 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.294167    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 08 13:50:48.305210 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.305185    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"]
Sep 08 13:50:48.440878 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.440845    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtk77\" (UniqueName: \"kubernetes.io/projected/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-api-access-wtk77\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.441089 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.440912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7a39f4-7b23-431d-8ad7-9440e544259d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.441089 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.440952    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.441089 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.440979    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.441251 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.441093    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7a39f4-7b23-431d-8ad7-9440e544259d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.542594 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.542516    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtk77\" (UniqueName: \"kubernetes.io/projected/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-api-access-wtk77\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.542594 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.542568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7a39f4-7b23-431d-8ad7-9440e544259d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.542594 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.542596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.542886 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.542613    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.542886 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.542641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7a39f4-7b23-431d-8ad7-9440e544259d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.542981 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.542964    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7a39f4-7b23-431d-8ad7-9440e544259d-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.543297 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.543274    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.543404 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.543375    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.545357 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.545340    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7a39f4-7b23-431d-8ad7-9440e544259d-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.550878 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.550849    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtk77\" (UniqueName: \"kubernetes.io/projected/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-api-access-wtk77\") pod \"isvc-tensorflow-predictor-547dfc5669-l47l9\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.602379 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.602355    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:48.878116 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.878087    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"]
Sep 08 13:50:48.880498 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:50:48.880472    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7a39f4_7b23_431d_8ad7_9440e544259d.slice/crio-8b4b31d6f48e83b8fc50bd3c37d10630567edbe730c65e8518acf7f64dc6b1c1 WatchSource:0}: Error finding container 8b4b31d6f48e83b8fc50bd3c37d10630567edbe730c65e8518acf7f64dc6b1c1: Status 404 returned error can't find the container with id 8b4b31d6f48e83b8fc50bd3c37d10630567edbe730c65e8518acf7f64dc6b1c1
Sep 08 13:50:48.886609 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:48.886575    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" event={"ID":"7a7a39f4-7b23-431d-8ad7-9440e544259d","Type":"ContainerStarted","Data":"8b4b31d6f48e83b8fc50bd3c37d10630567edbe730c65e8518acf7f64dc6b1c1"}
Sep 08 13:50:49.893227 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:49.893192    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" event={"ID":"7a7a39f4-7b23-431d-8ad7-9440e544259d","Type":"ContainerStarted","Data":"10641b1bb04c5047410a31f91145f1004c4f6577f30d53767520a82985b51e2f"}
Sep 08 13:50:49.955736 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:49.955694    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:50:51.838804 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:51.838769    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 08 13:50:53.914132 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:53.914091    2583 generic.go:356] "Generic (PLEG): container finished" podID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerID="10641b1bb04c5047410a31f91145f1004c4f6577f30d53767520a82985b51e2f" exitCode=0
Sep 08 13:50:53.914627 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:53.914166    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" event={"ID":"7a7a39f4-7b23-431d-8ad7-9440e544259d","Type":"ContainerDied","Data":"10641b1bb04c5047410a31f91145f1004c4f6577f30d53767520a82985b51e2f"}
Sep 08 13:50:53.915451 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:53.915435    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:50:57.937724 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:57.937629    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" event={"ID":"7a7a39f4-7b23-431d-8ad7-9440e544259d","Type":"ContainerStarted","Data":"50918f9dd158c9a682ccab0a5d7a903563b70e6fefcdedd045677b507b319659"}
Sep 08 13:50:57.937724 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:57.937706    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" event={"ID":"7a7a39f4-7b23-431d-8ad7-9440e544259d","Type":"ContainerStarted","Data":"e5fc7c737b546f46f365a93059c9f28b2bda8444a1e6fed5a28a0c5197df0897"}
Sep 08 13:50:57.938272 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:57.937993    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:57.938272 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:57.938059    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:50:57.939527 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:57.939499    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 08 13:50:57.961252 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:57.961204    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podStartSLOduration=1.943503347 podStartE2EDuration="9.9611899s" totalImagesPullingTime="3.65707563s" totalInitContainerRuntime="4.360610923s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:50:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:50:57.959374975 +0000 UTC m=+2086.888278717" watchObservedRunningTime="2026-09-08 13:50:57.9611899 +0000 UTC m=+2086.890093641"
Sep 08 13:50:58.942882 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:50:58.942840    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 08 13:51:01.839534 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:01.839490    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 08 13:51:03.947292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:03.947256    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:51:03.947909 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:03.947881    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 08 13:51:11.688912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:11.688879    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:51:11.689973 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:11.689943    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:51:11.838911 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:11.838870    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 08 13:51:13.948831 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:13.948798    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:51:21.839370 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:21.839332    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 08 13:51:28.948443 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:28.948411    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"]
Sep 08 13:51:28.948847 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:28.948649    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kserve-container" containerID="cri-o://e5fc7c737b546f46f365a93059c9f28b2bda8444a1e6fed5a28a0c5197df0897" gracePeriod=30
Sep 08 13:51:28.948847 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:28.948708    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy" containerID="cri-o://50918f9dd158c9a682ccab0a5d7a903563b70e6fefcdedd045677b507b319659" gracePeriod=30
Sep 08 13:51:28.978188 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:28.978148    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:29.079224 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:29.079193    2583 generic.go:356] "Generic (PLEG): container finished" podID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerID="50918f9dd158c9a682ccab0a5d7a903563b70e6fefcdedd045677b507b319659" exitCode=2
Sep 08 13:51:29.079357 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:29.079266    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" event={"ID":"7a7a39f4-7b23-431d-8ad7-9440e544259d","Type":"ContainerDied","Data":"50918f9dd158c9a682ccab0a5d7a903563b70e6fefcdedd045677b507b319659"}
Sep 08 13:51:31.839219 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:31.839181    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 08 13:51:33.943347 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:33.943309    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 08 13:51:34.095455 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:34.095418    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:34.181331 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:34.181293    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:38.943633 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:38.943591    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 08 13:51:41.838879 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:41.838840    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 08 13:51:43.943242 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:43.943199    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 08 13:51:43.943711 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:43.943330    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:51:48.943600 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:48.943551    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 08 13:51:50.425746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:50.425706    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"]
Sep 08 13:51:50.426223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:50.425976    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container" containerID="cri-o://799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b" gracePeriod=30
Sep 08 13:51:50.426223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:50.426055    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kube-rbac-proxy" containerID="cri-o://2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a" gracePeriod=30
Sep 08 13:51:50.427278 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:50.427063    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:51:51.182513 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:51.182476    2583 generic.go:356] "Generic (PLEG): container finished" podID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerID="2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a" exitCode=2
Sep 08 13:51:51.182693 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:51.182523    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" event={"ID":"45178f55-a6b8-4e0f-b846-1ed044e3f973","Type":"ContainerDied","Data":"2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a"}
Sep 08 13:51:51.831735 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:51.831691    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 08 13:51:53.943308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:53.943259    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 08 13:51:55.374390 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.374363    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:51:55.517785 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.517702    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-cabundle-cert\") pod \"45178f55-a6b8-4e0f-b846-1ed044e3f973\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") "
Sep 08 13:51:55.517785 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.517762    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45178f55-a6b8-4e0f-b846-1ed044e3f973-kserve-provision-location\") pod \"45178f55-a6b8-4e0f-b846-1ed044e3f973\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") "
Sep 08 13:51:55.518032 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.517807    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-rbac-proxy-sar-config\") pod \"45178f55-a6b8-4e0f-b846-1ed044e3f973\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") "
Sep 08 13:51:55.518032 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.517832    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45178f55-a6b8-4e0f-b846-1ed044e3f973-proxy-tls\") pod \"45178f55-a6b8-4e0f-b846-1ed044e3f973\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") "
Sep 08 13:51:55.518032 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.517864    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7czl7\" (UniqueName: \"kubernetes.io/projected/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-api-access-7czl7\") pod \"45178f55-a6b8-4e0f-b846-1ed044e3f973\" (UID: \"45178f55-a6b8-4e0f-b846-1ed044e3f973\") "
Sep 08 13:51:55.518219 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.518128    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45178f55-a6b8-4e0f-b846-1ed044e3f973-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45178f55-a6b8-4e0f-b846-1ed044e3f973" (UID: "45178f55-a6b8-4e0f-b846-1ed044e3f973"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:55.518219 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.518153    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45178f55-a6b8-4e0f-b846-1ed044e3f973" (UID: "45178f55-a6b8-4e0f-b846-1ed044e3f973"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:51:55.518219 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.518176    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45178f55-a6b8-4e0f-b846-1ed044e3f973" (UID: "45178f55-a6b8-4e0f-b846-1ed044e3f973"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:51:55.520251 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.520229    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45178f55-a6b8-4e0f-b846-1ed044e3f973-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45178f55-a6b8-4e0f-b846-1ed044e3f973" (UID: "45178f55-a6b8-4e0f-b846-1ed044e3f973"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:51:55.520316 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.520285    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-api-access-7czl7" (OuterVolumeSpecName: "kube-api-access-7czl7") pod "45178f55-a6b8-4e0f-b846-1ed044e3f973" (UID: "45178f55-a6b8-4e0f-b846-1ed044e3f973"). InnerVolumeSpecName "kube-api-access-7czl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:51:55.618816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.618785    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:51:55.618816 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.618813    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45178f55-a6b8-4e0f-b846-1ed044e3f973-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:51:55.619001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.618825    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:51:55.619001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.618835    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45178f55-a6b8-4e0f-b846-1ed044e3f973-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:51:55.619001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:55.618845    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7czl7\" (UniqueName: \"kubernetes.io/projected/45178f55-a6b8-4e0f-b846-1ed044e3f973-kube-api-access-7czl7\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:51:56.209671 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.209636    2583 generic.go:356] "Generic (PLEG): container finished" podID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerID="799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b" exitCode=0
Sep 08 13:51:56.209869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.209724    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"
Sep 08 13:51:56.209869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.209728    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" event={"ID":"45178f55-a6b8-4e0f-b846-1ed044e3f973","Type":"ContainerDied","Data":"799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b"}
Sep 08 13:51:56.209869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.209789    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb" event={"ID":"45178f55-a6b8-4e0f-b846-1ed044e3f973","Type":"ContainerDied","Data":"6cddbf4efe3c1e6b2e47d6cdf115836eb5383ff0c7e673ade1355e4d82f04320"}
Sep 08 13:51:56.210438 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.209822    2583 scope.go:117] "RemoveContainer" containerID="2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a"
Sep 08 13:51:56.219091 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.219075    2583 scope.go:117] "RemoveContainer" containerID="799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b"
Sep 08 13:51:56.227241 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.227159    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"]
Sep 08 13:51:56.227308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.227264    2583 scope.go:117] "RemoveContainer" containerID="1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850"
Sep 08 13:51:56.230648 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.230627    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-h9mrb"]
Sep 08 13:51:56.235401 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.235386    2583 scope.go:117] "RemoveContainer" containerID="2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a"
Sep 08 13:51:56.235701 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:51:56.235682    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a\": container with ID starting with 2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a not found: ID does not exist" containerID="2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a"
Sep 08 13:51:56.235753 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.235711    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a"} err="failed to get container status \"2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a\": rpc error: code = NotFound desc = could not find container \"2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a\": container with ID starting with 2f552eb8f496b6ed85a061958f1b67473e2b8362301e0ab007cbd78de8e13e5a not found: ID does not exist"
Sep 08 13:51:56.235753 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.235729    2583 scope.go:117] "RemoveContainer" containerID="799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b"
Sep 08 13:51:56.235992 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:51:56.235975    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b\": container with ID starting with 799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b not found: ID does not exist" containerID="799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b"
Sep 08 13:51:56.236084 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.236000    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b"} err="failed to get container status \"799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b\": rpc error: code = NotFound desc = could not find container \"799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b\": container with ID starting with 799c627ef04c522d9c74e4b0d30ff36501436017006516e2baaab070484ffb4b not found: ID does not exist"
Sep 08 13:51:56.236084 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.236036    2583 scope.go:117] "RemoveContainer" containerID="1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850"
Sep 08 13:51:56.236286 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:51:56.236269    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850\": container with ID starting with 1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850 not found: ID does not exist" containerID="1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850"
Sep 08 13:51:56.236325 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:56.236292    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850"} err="failed to get container status \"1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850\": rpc error: code = NotFound desc = could not find container \"1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850\": container with ID starting with 1f40e920019c772de993d176cca5e00904cea965616b106e2c0eaca88189a850 not found: ID does not exist"
Sep 08 13:51:57.595902 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:57.595856    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" path="/var/lib/kubelet/pods/45178f55-a6b8-4e0f-b846-1ed044e3f973/volumes"
Sep 08 13:51:58.293986 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.293954    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:51:58.385591 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.385554    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"]
Sep 08 13:51:58.385922 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.385910    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container"
Sep 08 13:51:58.385968 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.385924    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container"
Sep 08 13:51:58.385968 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.385939    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kube-rbac-proxy"
Sep 08 13:51:58.385968 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.385945    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kube-rbac-proxy"
Sep 08 13:51:58.385968 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.385958    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="storage-initializer"
Sep 08 13:51:58.385968 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.385964    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="storage-initializer"
Sep 08 13:51:58.386146 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.386028    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kube-rbac-proxy"
Sep 08 13:51:58.386146 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.386041    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="45178f55-a6b8-4e0f-b846-1ed044e3f973" containerName="kserve-container"
Sep 08 13:51:58.390837 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.390813    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.393661 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.393627    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 08 13:51:58.393983 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.393965    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 08 13:51:58.406701 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.406665    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"]
Sep 08 13:51:58.543111 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.543070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.543309 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.543135    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2859c304-1f6e-4104-ae60-d8e43c339d70-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.543309 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.543168    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.543309 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.543208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2859c304-1f6e-4104-ae60-d8e43c339d70-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.543309 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.543227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrh6w\" (UniqueName: \"kubernetes.io/projected/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-api-access-nrh6w\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.644244 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.644201    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2859c304-1f6e-4104-ae60-d8e43c339d70-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.644244 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.644246    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.644685 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.644280    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2859c304-1f6e-4104-ae60-d8e43c339d70-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.644685 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.644296    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrh6w\" (UniqueName: \"kubernetes.io/projected/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-api-access-nrh6w\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.644685 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.644363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.644685 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.644647    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2859c304-1f6e-4104-ae60-d8e43c339d70-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.644981 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.644960    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.645087 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.645067    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.647003 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.646982    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2859c304-1f6e-4104-ae60-d8e43c339d70-proxy-tls\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.654475 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.654450    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrh6w\" (UniqueName: \"kubernetes.io/projected/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-api-access-nrh6w\") pod \"isvc-canary-rollback-predictor-7cd6fd476f-r4fhr\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.701836 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.701793    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:51:58.943886 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.943795    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 08 13:51:58.982330 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:58.982234    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"]
Sep 08 13:51:58.984351 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:51:58.984326    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2859c304_1f6e_4104_ae60_d8e43c339d70.slice/crio-c89281bf85d6c8e790a884f6dd9ed278fdbd02db20380f01fadf41ec81f4ff76 WatchSource:0}: Error finding container c89281bf85d6c8e790a884f6dd9ed278fdbd02db20380f01fadf41ec81f4ff76: Status 404 returned error can't find the container with id c89281bf85d6c8e790a884f6dd9ed278fdbd02db20380f01fadf41ec81f4ff76
Sep 08 13:51:59.227744 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.227651    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" event={"ID":"2859c304-1f6e-4104-ae60-d8e43c339d70","Type":"ContainerStarted","Data":"d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d"}
Sep 08 13:51:59.227744 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.227725    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" event={"ID":"2859c304-1f6e-4104-ae60-d8e43c339d70","Type":"ContainerStarted","Data":"c89281bf85d6c8e790a884f6dd9ed278fdbd02db20380f01fadf41ec81f4ff76"}
Sep 08 13:51:59.230218 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.230192    2583 generic.go:356] "Generic (PLEG): container finished" podID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerID="e5fc7c737b546f46f365a93059c9f28b2bda8444a1e6fed5a28a0c5197df0897" exitCode=137
Sep 08 13:51:59.230352 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.230226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" event={"ID":"7a7a39f4-7b23-431d-8ad7-9440e544259d","Type":"ContainerDied","Data":"e5fc7c737b546f46f365a93059c9f28b2bda8444a1e6fed5a28a0c5197df0897"}
Sep 08 13:51:59.587245 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.587212    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:51:59.754891 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.754858    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtk77\" (UniqueName: \"kubernetes.io/projected/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-api-access-wtk77\") pod \"7a7a39f4-7b23-431d-8ad7-9440e544259d\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") "
Sep 08 13:51:59.754891 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.754895    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7a39f4-7b23-431d-8ad7-9440e544259d-proxy-tls\") pod \"7a7a39f4-7b23-431d-8ad7-9440e544259d\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") "
Sep 08 13:51:59.755457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.754954    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7a39f4-7b23-431d-8ad7-9440e544259d-kserve-provision-location\") pod \"7a7a39f4-7b23-431d-8ad7-9440e544259d\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") "
Sep 08 13:51:59.755457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.755035    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-cabundle-cert\") pod \"7a7a39f4-7b23-431d-8ad7-9440e544259d\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") "
Sep 08 13:51:59.755457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.755069    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-rbac-proxy-sar-config\") pod \"7a7a39f4-7b23-431d-8ad7-9440e544259d\" (UID: \"7a7a39f4-7b23-431d-8ad7-9440e544259d\") "
Sep 08 13:51:59.755457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.755381    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a7a39f4-7b23-431d-8ad7-9440e544259d" (UID: "7a7a39f4-7b23-431d-8ad7-9440e544259d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:51:59.755649 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.755476    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a7a39f4-7b23-431d-8ad7-9440e544259d" (UID: "7a7a39f4-7b23-431d-8ad7-9440e544259d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:51:59.757586 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.757549    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-api-access-wtk77" (OuterVolumeSpecName: "kube-api-access-wtk77") pod "7a7a39f4-7b23-431d-8ad7-9440e544259d" (UID: "7a7a39f4-7b23-431d-8ad7-9440e544259d"). InnerVolumeSpecName "kube-api-access-wtk77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:51:59.757586 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.757578    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7a39f4-7b23-431d-8ad7-9440e544259d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a7a39f4-7b23-431d-8ad7-9440e544259d" (UID: "7a7a39f4-7b23-431d-8ad7-9440e544259d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:51:59.766552 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.766521    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a7a39f4-7b23-431d-8ad7-9440e544259d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a7a39f4-7b23-431d-8ad7-9440e544259d" (UID: "7a7a39f4-7b23-431d-8ad7-9440e544259d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:51:59.856313 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.856281    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:51:59.856313 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.856311    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtk77\" (UniqueName: \"kubernetes.io/projected/7a7a39f4-7b23-431d-8ad7-9440e544259d-kube-api-access-wtk77\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:51:59.856519 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.856321    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a7a39f4-7b23-431d-8ad7-9440e544259d-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:51:59.856519 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.856332    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a7a39f4-7b23-431d-8ad7-9440e544259d-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:51:59.856519 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:51:59.856341    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a7a39f4-7b23-431d-8ad7-9440e544259d-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:52:00.237351 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:00.237264    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"
Sep 08 13:52:00.237351 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:00.237281    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9" event={"ID":"7a7a39f4-7b23-431d-8ad7-9440e544259d","Type":"ContainerDied","Data":"8b4b31d6f48e83b8fc50bd3c37d10630567edbe730c65e8518acf7f64dc6b1c1"}
Sep 08 13:52:00.237599 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:00.237357    2583 scope.go:117] "RemoveContainer" containerID="50918f9dd158c9a682ccab0a5d7a903563b70e6fefcdedd045677b507b319659"
Sep 08 13:52:00.246454 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:00.246435    2583 scope.go:117] "RemoveContainer" containerID="e5fc7c737b546f46f365a93059c9f28b2bda8444a1e6fed5a28a0c5197df0897"
Sep 08 13:52:00.254154 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:00.254136    2583 scope.go:117] "RemoveContainer" containerID="10641b1bb04c5047410a31f91145f1004c4f6577f30d53767520a82985b51e2f"
Sep 08 13:52:00.263985 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:00.263952    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"]
Sep 08 13:52:00.269162 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:00.269140    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-l47l9"]
Sep 08 13:52:01.595254 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:01.595219    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" path="/var/lib/kubelet/pods/7a7a39f4-7b23-431d-8ad7-9440e544259d/volumes"
Sep 08 13:52:03.254454 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:03.254415    2583 generic.go:356] "Generic (PLEG): container finished" podID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerID="d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d" exitCode=0
Sep 08 13:52:03.254912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:03.254491    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" event={"ID":"2859c304-1f6e-4104-ae60-d8e43c339d70","Type":"ContainerDied","Data":"d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d"}
Sep 08 13:52:04.261229 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:04.261192    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" event={"ID":"2859c304-1f6e-4104-ae60-d8e43c339d70","Type":"ContainerStarted","Data":"f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618"}
Sep 08 13:52:04.261635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:04.261282    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" event={"ID":"2859c304-1f6e-4104-ae60-d8e43c339d70","Type":"ContainerStarted","Data":"c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8"}
Sep 08 13:52:04.261635 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:04.261607    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:52:04.261765 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:04.261748    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:52:04.263182 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:04.263153    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:52:04.293184 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:04.293133    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podStartSLOduration=2.286219285 podStartE2EDuration="6.293114972s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.006895687s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:52:04.291542876 +0000 UTC m=+2153.220446634" watchObservedRunningTime="2026-09-08 13:52:04.293114972 +0000 UTC m=+2153.222018714"
Sep 08 13:52:05.266947 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:05.266903    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:52:10.271990 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:10.271951    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:52:10.272642 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:10.272616    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:52:16.534044 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:16.533991    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:20.273278 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:20.273235    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:52:20.330245 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:20.330204    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:20.439648 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:20.439613    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:52:30.273431 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:30.273387    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:52:40.272766 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:40.272727    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:52:50.273522 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:52:50.273475    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:53:00.273224 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:00.273176    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:53:06.985928 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:06.985893    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:10.272607 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:10.272557    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:53:20.274054 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:20.274001    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:53:28.457705 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:28.457672    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"]
Sep 08 13:53:28.458110 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:28.457902    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" containerID="cri-o://c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8" gracePeriod=30
Sep 08 13:53:28.458110 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:28.457941    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kube-rbac-proxy" containerID="cri-o://f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618" gracePeriod=30
Sep 08 13:53:28.484243 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:28.484211    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:28.652645 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:28.652611    2583 generic.go:356] "Generic (PLEG): container finished" podID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerID="f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618" exitCode=2
Sep 08 13:53:28.652801 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:28.652654    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" event={"ID":"2859c304-1f6e-4104-ae60-d8e43c339d70","Type":"ContainerDied","Data":"f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618"}
Sep 08 13:53:30.267826 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:30.267787    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 08 13:53:30.273186 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:30.273161    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 08 13:53:33.497879 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.497854    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:53:33.534526 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.534490    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrh6w\" (UniqueName: \"kubernetes.io/projected/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-api-access-nrh6w\") pod \"2859c304-1f6e-4104-ae60-d8e43c339d70\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") "
Sep 08 13:53:33.534699 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.534577    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2859c304-1f6e-4104-ae60-d8e43c339d70-kserve-provision-location\") pod \"2859c304-1f6e-4104-ae60-d8e43c339d70\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") "
Sep 08 13:53:33.534699 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.534608    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-cabundle-cert\") pod \"2859c304-1f6e-4104-ae60-d8e43c339d70\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") "
Sep 08 13:53:33.534699 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.534673    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-rbac-proxy-sar-config\") pod \"2859c304-1f6e-4104-ae60-d8e43c339d70\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") "
Sep 08 13:53:33.534875 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.534737    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2859c304-1f6e-4104-ae60-d8e43c339d70-proxy-tls\") pod \"2859c304-1f6e-4104-ae60-d8e43c339d70\" (UID: \"2859c304-1f6e-4104-ae60-d8e43c339d70\") "
Sep 08 13:53:33.534943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.534915    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2859c304-1f6e-4104-ae60-d8e43c339d70-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2859c304-1f6e-4104-ae60-d8e43c339d70" (UID: "2859c304-1f6e-4104-ae60-d8e43c339d70"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:53:33.534998 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.534987    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2859c304-1f6e-4104-ae60-d8e43c339d70-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:53:33.535071 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.535053    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2859c304-1f6e-4104-ae60-d8e43c339d70" (UID: "2859c304-1f6e-4104-ae60-d8e43c339d70"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:53:33.535157 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.535127    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2859c304-1f6e-4104-ae60-d8e43c339d70" (UID: "2859c304-1f6e-4104-ae60-d8e43c339d70"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:53:33.536844 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.536811    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-api-access-nrh6w" (OuterVolumeSpecName: "kube-api-access-nrh6w") pod "2859c304-1f6e-4104-ae60-d8e43c339d70" (UID: "2859c304-1f6e-4104-ae60-d8e43c339d70"). InnerVolumeSpecName "kube-api-access-nrh6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:53:33.536945 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.536859    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2859c304-1f6e-4104-ae60-d8e43c339d70-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2859c304-1f6e-4104-ae60-d8e43c339d70" (UID: "2859c304-1f6e-4104-ae60-d8e43c339d70"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:53:33.641139 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.636481    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrh6w\" (UniqueName: \"kubernetes.io/projected/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-api-access-nrh6w\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:53:33.641139 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.636513    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:53:33.641139 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.636528    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2859c304-1f6e-4104-ae60-d8e43c339d70-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:53:33.641139 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.636549    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2859c304-1f6e-4104-ae60-d8e43c339d70-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:53:33.655446 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.655415    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:33.677787 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.677756    2583 generic.go:356] "Generic (PLEG): container finished" podID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerID="c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8" exitCode=0
Sep 08 13:53:33.677923 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.677833    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" event={"ID":"2859c304-1f6e-4104-ae60-d8e43c339d70","Type":"ContainerDied","Data":"c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8"}
Sep 08 13:53:33.677923 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.677851    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"
Sep 08 13:53:33.678108 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.677923    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr" event={"ID":"2859c304-1f6e-4104-ae60-d8e43c339d70","Type":"ContainerDied","Data":"c89281bf85d6c8e790a884f6dd9ed278fdbd02db20380f01fadf41ec81f4ff76"}
Sep 08 13:53:33.678108 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.677954    2583 scope.go:117] "RemoveContainer" containerID="f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618"
Sep 08 13:53:33.688342 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.688321    2583 scope.go:117] "RemoveContainer" containerID="c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8"
Sep 08 13:53:33.696251 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.696232    2583 scope.go:117] "RemoveContainer" containerID="d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d"
Sep 08 13:53:33.704805 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.704775    2583 scope.go:117] "RemoveContainer" containerID="f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618"
Sep 08 13:53:33.705167 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:53:33.705142    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618\": container with ID starting with f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618 not found: ID does not exist" containerID="f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618"
Sep 08 13:53:33.705263 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.705176    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618"} err="failed to get container status \"f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618\": rpc error: code = NotFound desc = could not find container \"f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618\": container with ID starting with f6c6d04b40d851a28fb8f37030f0cd304b97e77cb610597423985d6b9a8ce618 not found: ID does not exist"
Sep 08 13:53:33.705263 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.705194    2583 scope.go:117] "RemoveContainer" containerID="c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8"
Sep 08 13:53:33.705520 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:53:33.705497    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8\": container with ID starting with c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8 not found: ID does not exist" containerID="c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8"
Sep 08 13:53:33.705599 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.705527    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8"} err="failed to get container status \"c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8\": rpc error: code = NotFound desc = could not find container \"c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8\": container with ID starting with c9dbfe1a9d29a7c89ed8dad9ae0d757e2420c5fd5ce864376a34cb36eebee5a8 not found: ID does not exist"
Sep 08 13:53:33.705599 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.705548    2583 scope.go:117] "RemoveContainer" containerID="d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d"
Sep 08 13:53:33.705852 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:53:33.705837    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d\": container with ID starting with d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d not found: ID does not exist" containerID="d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d"
Sep 08 13:53:33.705921 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.705855    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d"} err="failed to get container status \"d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d\": rpc error: code = NotFound desc = could not find container \"d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d\": container with ID starting with d55f5c7d4f730d8fb181cfae7a9b48f4fd7347d4f0332260e6f00ebefe109e0d not found: ID does not exist"
Sep 08 13:53:33.726575 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.726280    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"]
Sep 08 13:53:33.739591 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.739567    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"]
Sep 08 13:53:33.740116 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740099    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy"
Sep 08 13:53:33.740204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740118    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy"
Sep 08 13:53:33.740204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740136    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="storage-initializer"
Sep 08 13:53:33.740204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740145    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="storage-initializer"
Sep 08 13:53:33.740204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740157    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kserve-container"
Sep 08 13:53:33.740204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740166    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kserve-container"
Sep 08 13:53:33.740204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740174    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container"
Sep 08 13:53:33.740204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740184    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container"
Sep 08 13:53:33.740204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740193    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kube-rbac-proxy"
Sep 08 13:53:33.740204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740201    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kube-rbac-proxy"
Sep 08 13:53:33.740558 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740226    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="storage-initializer"
Sep 08 13:53:33.740558 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740234    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="storage-initializer"
Sep 08 13:53:33.740558 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740302    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kube-rbac-proxy"
Sep 08 13:53:33.740558 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740313    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kube-rbac-proxy"
Sep 08 13:53:33.740558 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740323    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" containerName="kserve-container"
Sep 08 13:53:33.740558 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.740334    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a7a39f4-7b23-431d-8ad7-9440e544259d" containerName="kserve-container"
Sep 08 13:53:33.745224 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.745205    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7cd6fd476f-r4fhr"]
Sep 08 13:53:33.745335 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.745319    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.747689 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.747671    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 08 13:53:33.748283 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.748269    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:53:33.748666 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.748646    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:53:33.748666 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.748657    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 08 13:53:33.748830 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.748750    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:53:33.749187 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.749168    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:53:33.752857 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.752829    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:53:33.760428 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.760363    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:33.762634 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.762566    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"]
Sep 08 13:53:33.839045 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.838995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.839212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.839083    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.839212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.839127    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.839212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.839159    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.839212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.839177    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbv2\" (UniqueName: \"kubernetes.io/projected/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-api-access-tsbv2\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.865504 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.865482    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"]
Sep 08 13:53:33.868796 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.868781    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:33.871691 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.871672    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 08 13:53:33.887346 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.887321    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"]
Sep 08 13:53:33.939716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.939642    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.939716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.939708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.939915 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.939735    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbv2\" (UniqueName: \"kubernetes.io/projected/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-api-access-tsbv2\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.939915 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.939769    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqs4l\" (UniqueName: \"kubernetes.io/projected/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-api-access-xqs4l\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:33.939915 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.939802    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.940107 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.939992    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:33.940107 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.940049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:33.940212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.940110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d511110a-9bde-4fc9-83dd-415eaede9cfa-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:33.940212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.940129    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d511110a-9bde-4fc9-83dd-415eaede9cfa-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:33.940212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.940170    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.940487 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.940471    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.940617 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.940597    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.940675 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.940650    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.942377 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.942359    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:33.953271 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:33.953246    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbv2\" (UniqueName: \"kubernetes.io/projected/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-api-access-tsbv2\") pod \"isvc-canary-stop-v2-predictor-56ddb6965b-thhl6\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:34.041362 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.041334    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.041362 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.041363    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.041556 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.041392    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d511110a-9bde-4fc9-83dd-415eaede9cfa-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.041556 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.041425    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d511110a-9bde-4fc9-83dd-415eaede9cfa-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.041556 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.041503    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqs4l\" (UniqueName: \"kubernetes.io/projected/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-api-access-xqs4l\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.041756 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.041737    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d511110a-9bde-4fc9-83dd-415eaede9cfa-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.042105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.042085    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.042180 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.042100    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-cabundle-cert\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.043931 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.043901    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d511110a-9bde-4fc9-83dd-415eaede9cfa-proxy-tls\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.053824 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.053795    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqs4l\" (UniqueName: \"kubernetes.io/projected/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-api-access-xqs4l\") pod \"isvc-canary-stop-predictor-7b7754b5bf-kktr7\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.056388 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.056366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:34.179264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.179233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:34.341738 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.341713    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"]
Sep 08 13:53:34.343639 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:53:34.343611    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cdb538_3c8f_4996_b7d5_f9ad1205b9cc.slice/crio-f5b286baf68ce7a157f84621e4032eab592a3feeda9a3e2f5f57ec418ce68e23 WatchSource:0}: Error finding container f5b286baf68ce7a157f84621e4032eab592a3feeda9a3e2f5f57ec418ce68e23: Status 404 returned error can't find the container with id f5b286baf68ce7a157f84621e4032eab592a3feeda9a3e2f5f57ec418ce68e23
Sep 08 13:53:34.454849 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.454821    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"]
Sep 08 13:53:34.457932 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:53:34.457909    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd511110a_9bde_4fc9_83dd_415eaede9cfa.slice/crio-b723176afe3b254bc9227f8759e735137cbd6f51cfb7f0e992fd55e053184a23 WatchSource:0}: Error finding container b723176afe3b254bc9227f8759e735137cbd6f51cfb7f0e992fd55e053184a23: Status 404 returned error can't find the container with id b723176afe3b254bc9227f8759e735137cbd6f51cfb7f0e992fd55e053184a23
Sep 08 13:53:34.586972 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.586935    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:34.686341 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.686302    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" event={"ID":"d511110a-9bde-4fc9-83dd-415eaede9cfa","Type":"ContainerStarted","Data":"82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a"}
Sep 08 13:53:34.686512 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.686378    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" event={"ID":"d511110a-9bde-4fc9-83dd-415eaede9cfa","Type":"ContainerStarted","Data":"b723176afe3b254bc9227f8759e735137cbd6f51cfb7f0e992fd55e053184a23"}
Sep 08 13:53:34.688094 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.688072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" event={"ID":"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc","Type":"ContainerStarted","Data":"9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80"}
Sep 08 13:53:34.688221 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.688125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" event={"ID":"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc","Type":"ContainerStarted","Data":"f5b286baf68ce7a157f84621e4032eab592a3feeda9a3e2f5f57ec418ce68e23"}
Sep 08 13:53:34.691253 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:34.691232    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:53:35.595365 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:35.595332    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2859c304-1f6e-4104-ae60-d8e43c339d70" path="/var/lib/kubelet/pods/2859c304-1f6e-4104-ae60-d8e43c339d70/volumes"
Sep 08 13:53:38.705723 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:38.705687    2583 generic.go:356] "Generic (PLEG): container finished" podID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerID="82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a" exitCode=0
Sep 08 13:53:38.706150 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:38.705759    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" event={"ID":"d511110a-9bde-4fc9-83dd-415eaede9cfa","Type":"ContainerDied","Data":"82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a"}
Sep 08 13:53:38.707361 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:38.707338    2583 generic.go:356] "Generic (PLEG): container finished" podID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerID="9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80" exitCode=0
Sep 08 13:53:38.707453 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:38.707407    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" event={"ID":"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc","Type":"ContainerDied","Data":"9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80"}
Sep 08 13:53:39.715212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.715179    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" event={"ID":"d511110a-9bde-4fc9-83dd-415eaede9cfa","Type":"ContainerStarted","Data":"4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5"}
Sep 08 13:53:39.715700 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.715257    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" event={"ID":"d511110a-9bde-4fc9-83dd-415eaede9cfa","Type":"ContainerStarted","Data":"5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39"}
Sep 08 13:53:39.715700 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.715478    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:39.717629 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.717609    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" event={"ID":"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc","Type":"ContainerStarted","Data":"0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca"}
Sep 08 13:53:39.717723 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.717649    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" event={"ID":"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc","Type":"ContainerStarted","Data":"0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1"}
Sep 08 13:53:39.717999 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.717980    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:39.718091 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.718056    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:39.719741 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.719713    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:53:39.740444 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.740399    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podStartSLOduration=2.989040193 podStartE2EDuration="6.740385546s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.751345353s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:53:39.737847846 +0000 UTC m=+2248.666751608" watchObservedRunningTime="2026-09-08 13:53:39.740385546 +0000 UTC m=+2248.669289290"
Sep 08 13:53:39.762171 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:39.762122    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podStartSLOduration=2.983147028 podStartE2EDuration="6.762109804s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.778962776s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:53:39.761350869 +0000 UTC m=+2248.690254639" watchObservedRunningTime="2026-09-08 13:53:39.762109804 +0000 UTC m=+2248.691013571"
Sep 08 13:53:40.722827 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:40.722792    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:53:40.723304 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:40.722932    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:40.724007 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:40.723982    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:53:41.726820 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:41.726788    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:53:45.727512 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:45.727485    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:53:45.727988 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:45.727960    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:53:46.731928 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:46.731902    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:53:46.732437 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:46.732412    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:53:55.728810 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:55.728772    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:53:56.732585 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:53:56.732539    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:54:05.728882 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:05.728844    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:54:06.733218 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:06.733180    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:54:15.728484 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:15.728444    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:54:16.732714 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:16.732678    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:54:25.728055 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:25.727996    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:54:26.732685 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:26.732646    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:54:32.754212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:32.754174    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:54:35.728659 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:35.728621    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:54:36.733347 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:36.733310    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:54:37.899475 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:37.899439    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:54:38.052317 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:38.052282    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:54:45.728907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:45.728868    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:54:46.733167 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:46.733129    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:54:49.601504 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:49.601457    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 08 13:54:55.595257 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:55.595230    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:54:59.602158 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:54:59.602130    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:55:03.702838 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:03.702751    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"]
Sep 08 13:55:03.703315 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:03.703031    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container" containerID="cri-o://0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1" gracePeriod=30
Sep 08 13:55:03.703388 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:03.703349    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kube-rbac-proxy" containerID="cri-o://0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca" gracePeriod=30
Sep 08 13:55:03.726912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:03.726880    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:03.761845 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:03.761814    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"]
Sep 08 13:55:03.762099 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:03.762061    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" containerID="cri-o://5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39" gracePeriod=30
Sep 08 13:55:03.762099 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:03.762085    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kube-rbac-proxy" containerID="cri-o://4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5" gracePeriod=30
Sep 08 13:55:03.817400 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:03.817358    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:04.117518 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:04.117483    2583 generic.go:356] "Generic (PLEG): container finished" podID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerID="4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5" exitCode=2
Sep 08 13:55:04.117718 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:04.117566    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" event={"ID":"d511110a-9bde-4fc9-83dd-415eaede9cfa","Type":"ContainerDied","Data":"4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5"}
Sep 08 13:55:04.119968 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:04.119932    2583 generic.go:356] "Generic (PLEG): container finished" podID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerID="0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca" exitCode=2
Sep 08 13:55:04.120125 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:04.120036    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" event={"ID":"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc","Type":"ContainerDied","Data":"0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca"}
Sep 08 13:55:05.590882 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:05.590834    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 08 13:55:05.723212 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:05.723169    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 08 13:55:06.727935 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:06.727896    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 08 13:55:08.752620 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.752589    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:55:08.815960 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.815931    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:55:08.857976 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.857925    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xqs4l\" (UniqueName: \"kubernetes.io/projected/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-api-access-xqs4l\") pod \"d511110a-9bde-4fc9-83dd-415eaede9cfa\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") "
Sep 08 13:55:08.857976 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.857983    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d511110a-9bde-4fc9-83dd-415eaede9cfa-kserve-provision-location\") pod \"d511110a-9bde-4fc9-83dd-415eaede9cfa\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") "
Sep 08 13:55:08.858261 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858039    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-rbac-proxy-sar-config\") pod \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") "
Sep 08 13:55:08.858261 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858177    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-rbac-proxy-sar-config\") pod \"d511110a-9bde-4fc9-83dd-415eaede9cfa\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") "
Sep 08 13:55:08.858261 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858236    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kserve-provision-location\") pod \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") "
Sep 08 13:55:08.858424 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858316    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsbv2\" (UniqueName: \"kubernetes.io/projected/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-api-access-tsbv2\") pod \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") "
Sep 08 13:55:08.858487 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858351    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d511110a-9bde-4fc9-83dd-415eaede9cfa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d511110a-9bde-4fc9-83dd-415eaede9cfa" (UID: "d511110a-9bde-4fc9-83dd-415eaede9cfa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:55:08.858487 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858362    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-cabundle-cert\") pod \"d511110a-9bde-4fc9-83dd-415eaede9cfa\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") "
Sep 08 13:55:08.858604 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858485    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d511110a-9bde-4fc9-83dd-415eaede9cfa-proxy-tls\") pod \"d511110a-9bde-4fc9-83dd-415eaede9cfa\" (UID: \"d511110a-9bde-4fc9-83dd-415eaede9cfa\") "
Sep 08 13:55:08.858604 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858486    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" (UID: "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:55:08.858604 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858518    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-cabundle-cert\") pod \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") "
Sep 08 13:55:08.858604 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-proxy-tls\") pod \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\" (UID: \"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc\") "
Sep 08 13:55:08.858604 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858589    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d511110a-9bde-4fc9-83dd-415eaede9cfa" (UID: "d511110a-9bde-4fc9-83dd-415eaede9cfa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:55:08.858868 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858636    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" (UID: "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:55:08.859043 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.858988    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" (UID: "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:55:08.859292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.859056    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d511110a-9bde-4fc9-83dd-415eaede9cfa" (UID: "d511110a-9bde-4fc9-83dd-415eaede9cfa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:55:08.859292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.859128    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:08.859292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.859151    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d511110a-9bde-4fc9-83dd-415eaede9cfa-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:08.859292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.859168    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:08.859292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.859182    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:08.859292 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.859198    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:08.860417 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.860395    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-api-access-xqs4l" (OuterVolumeSpecName: "kube-api-access-xqs4l") pod "d511110a-9bde-4fc9-83dd-415eaede9cfa" (UID: "d511110a-9bde-4fc9-83dd-415eaede9cfa"). InnerVolumeSpecName "kube-api-access-xqs4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:55:08.860640 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.860626    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d511110a-9bde-4fc9-83dd-415eaede9cfa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d511110a-9bde-4fc9-83dd-415eaede9cfa" (UID: "d511110a-9bde-4fc9-83dd-415eaede9cfa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:55:08.861285 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.861259    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" (UID: "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:55:08.861285 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.861269    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-api-access-tsbv2" (OuterVolumeSpecName: "kube-api-access-tsbv2") pod "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" (UID: "40cdb538-3c8f-4996-b7d5-f9ad1205b9cc"). InnerVolumeSpecName "kube-api-access-tsbv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:55:08.960587 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.960547    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsbv2\" (UniqueName: \"kubernetes.io/projected/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-kube-api-access-tsbv2\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:08.960587 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.960580    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d511110a-9bde-4fc9-83dd-415eaede9cfa-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:08.960587 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.960590    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d511110a-9bde-4fc9-83dd-415eaede9cfa-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:08.960587 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.960598    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:08.960856 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:08.960607    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqs4l\" (UniqueName: \"kubernetes.io/projected/d511110a-9bde-4fc9-83dd-415eaede9cfa-kube-api-access-xqs4l\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:55:09.143625 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.143585    2583 generic.go:356] "Generic (PLEG): container finished" podID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerID="5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39" exitCode=0
Sep 08 13:55:09.143823 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.143653    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"
Sep 08 13:55:09.143823 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.143664    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" event={"ID":"d511110a-9bde-4fc9-83dd-415eaede9cfa","Type":"ContainerDied","Data":"5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39"}
Sep 08 13:55:09.143823 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.143737    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7" event={"ID":"d511110a-9bde-4fc9-83dd-415eaede9cfa","Type":"ContainerDied","Data":"b723176afe3b254bc9227f8759e735137cbd6f51cfb7f0e992fd55e053184a23"}
Sep 08 13:55:09.143823 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.143778    2583 scope.go:117] "RemoveContainer" containerID="4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5"
Sep 08 13:55:09.145983 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.145956    2583 generic.go:356] "Generic (PLEG): container finished" podID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerID="0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1" exitCode=0
Sep 08 13:55:09.146128 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.146001    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"
Sep 08 13:55:09.146128 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.146063    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" event={"ID":"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc","Type":"ContainerDied","Data":"0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1"}
Sep 08 13:55:09.146254 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.146132    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6" event={"ID":"40cdb538-3c8f-4996-b7d5-f9ad1205b9cc","Type":"ContainerDied","Data":"f5b286baf68ce7a157f84621e4032eab592a3feeda9a3e2f5f57ec418ce68e23"}
Sep 08 13:55:09.154192 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.153967    2583 scope.go:117] "RemoveContainer" containerID="5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39"
Sep 08 13:55:09.161695 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.161677    2583 scope.go:117] "RemoveContainer" containerID="82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a"
Sep 08 13:55:09.169197 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.169177    2583 scope.go:117] "RemoveContainer" containerID="4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5"
Sep 08 13:55:09.169479 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:55:09.169459    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5\": container with ID starting with 4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5 not found: ID does not exist" containerID="4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5"
Sep 08 13:55:09.169552 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.169486    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5"} err="failed to get container status \"4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5\": rpc error: code = NotFound desc = could not find container \"4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5\": container with ID starting with 4ad49e73e0bf525b90ab7957e04e6aa94d387be7cd08683aed04a2b0f1c9d5c5 not found: ID does not exist"
Sep 08 13:55:09.169552 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.169503    2583 scope.go:117] "RemoveContainer" containerID="5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39"
Sep 08 13:55:09.169803 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:55:09.169777    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39\": container with ID starting with 5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39 not found: ID does not exist" containerID="5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39"
Sep 08 13:55:09.169854 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.169810    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39"} err="failed to get container status \"5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39\": rpc error: code = NotFound desc = could not find container \"5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39\": container with ID starting with 5dc947d167648a42f7c6b63c47e84352dd27744a2968184c0fd4d1d664e33b39 not found: ID does not exist"
Sep 08 13:55:09.169854 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.169826    2583 scope.go:117] "RemoveContainer" containerID="82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a"
Sep 08 13:55:09.170098 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:55:09.170082    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a\": container with ID starting with 82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a not found: ID does not exist" containerID="82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a"
Sep 08 13:55:09.170159 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.170106    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a"} err="failed to get container status \"82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a\": rpc error: code = NotFound desc = could not find container \"82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a\": container with ID starting with 82d801bca7bf3de5e4866863e9c8abbe6c31d05cc7537a70de8f90301f7f5e7a not found: ID does not exist"
Sep 08 13:55:09.170159 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.170125    2583 scope.go:117] "RemoveContainer" containerID="0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca"
Sep 08 13:55:09.177399 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.177384    2583 scope.go:117] "RemoveContainer" containerID="0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1"
Sep 08 13:55:09.185950 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.185872    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"]
Sep 08 13:55:09.185950 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.185895    2583 scope.go:117] "RemoveContainer" containerID="9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80"
Sep 08 13:55:09.188505 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.188487    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-56ddb6965b-thhl6"]
Sep 08 13:55:09.194117 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.194102    2583 scope.go:117] "RemoveContainer" containerID="0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca"
Sep 08 13:55:09.194374 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:55:09.194351    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca\": container with ID starting with 0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca not found: ID does not exist" containerID="0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca"
Sep 08 13:55:09.194418 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.194381    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca"} err="failed to get container status \"0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca\": rpc error: code = NotFound desc = could not find container \"0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca\": container with ID starting with 0cf51ac9fb003fa0247d643db831ad140fcf0350e7ed293770c3365f1e8d21ca not found: ID does not exist"
Sep 08 13:55:09.194418 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.194399    2583 scope.go:117] "RemoveContainer" containerID="0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1"
Sep 08 13:55:09.194684 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:55:09.194665    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1\": container with ID starting with 0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1 not found: ID does not exist" containerID="0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1"
Sep 08 13:55:09.194720 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.194691    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1"} err="failed to get container status \"0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1\": rpc error: code = NotFound desc = could not find container \"0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1\": container with ID starting with 0aa9ede6c3b461452d4acca4750d6e78583ca864c0c63da08b30e7c4d24869d1 not found: ID does not exist"
Sep 08 13:55:09.194720 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.194706    2583 scope.go:117] "RemoveContainer" containerID="9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80"
Sep 08 13:55:09.194971 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:55:09.194948    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80\": container with ID starting with 9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80 not found: ID does not exist" containerID="9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80"
Sep 08 13:55:09.195393 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.194973    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80"} err="failed to get container status \"9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80\": rpc error: code = NotFound desc = could not find container \"9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80\": container with ID starting with 9a562d89b8f07cfe1b1c498510a317c9b96d524a3c965eca87c34013bcc35d80 not found: ID does not exist"
Sep 08 13:55:09.203196 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.203174    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"]
Sep 08 13:55:09.207179 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.207159    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7b7754b5bf-kktr7"]
Sep 08 13:55:09.595533 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.595498    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" path="/var/lib/kubelet/pods/40cdb538-3c8f-4996-b7d5-f9ad1205b9cc/volumes"
Sep 08 13:55:09.595978 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:09.595964    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" path="/var/lib/kubelet/pods/d511110a-9bde-4fc9-83dd-415eaede9cfa/volumes"
Sep 08 13:55:11.345086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:11.345048    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:14.067501 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.067462    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:14.093119 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093085    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"]
Sep 08 13:55:14.093416 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093403    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="storage-initializer"
Sep 08 13:55:14.093474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093418    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="storage-initializer"
Sep 08 13:55:14.093474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093428    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kube-rbac-proxy"
Sep 08 13:55:14.093474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093433    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kube-rbac-proxy"
Sep 08 13:55:14.093474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093439    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container"
Sep 08 13:55:14.093474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093445    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container"
Sep 08 13:55:14.093474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093455    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="storage-initializer"
Sep 08 13:55:14.093474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093460    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="storage-initializer"
Sep 08 13:55:14.093474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093471    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container"
Sep 08 13:55:14.093474 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093476    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container"
Sep 08 13:55:14.093902 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093483    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kube-rbac-proxy"
Sep 08 13:55:14.093902 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093487    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kube-rbac-proxy"
Sep 08 13:55:14.093902 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093535    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kserve-container"
Sep 08 13:55:14.093902 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093544    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kserve-container"
Sep 08 13:55:14.093902 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093550    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d511110a-9bde-4fc9-83dd-415eaede9cfa" containerName="kube-rbac-proxy"
Sep 08 13:55:14.093902 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.093556    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="40cdb538-3c8f-4996-b7d5-f9ad1205b9cc" containerName="kube-rbac-proxy"
Sep 08 13:55:14.098777 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.098755    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.106521 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.106495    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 08 13:55:14.106521 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.106509    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 13:55:14.106697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.106573    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 08 13:55:14.106784 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.106768    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 13:55:14.107077 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.107060    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 13:55:14.107473 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.107458    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 13:55:14.107560 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.107485    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 13:55:14.118774 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.118752    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"]
Sep 08 13:55:14.204733 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.204695    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.204733 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.204729    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nxfz\" (UniqueName: \"kubernetes.io/projected/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-api-access-5nxfz\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.204943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.204797    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.204943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.204890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dca1633-3fb5-4cab-8d70-e996118b2f33-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.204943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.204923    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dca1633-3fb5-4cab-8d70-e996118b2f33-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.222394 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.222363    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:55:14.305587 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.305551    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.305587 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.305589    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nxfz\" (UniqueName: \"kubernetes.io/projected/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-api-access-5nxfz\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.305869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.305763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.305869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.305833    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dca1633-3fb5-4cab-8d70-e996118b2f33-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.305869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.305863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dca1633-3fb5-4cab-8d70-e996118b2f33-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.306240 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.306207    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dca1633-3fb5-4cab-8d70-e996118b2f33-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.306371 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.306353    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.306429 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.306368    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.308510 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.308492    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dca1633-3fb5-4cab-8d70-e996118b2f33-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.317161 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.317141    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nxfz\" (UniqueName: \"kubernetes.io/projected/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-api-access-5nxfz\") pod \"isvc-lightgbm-predictor-54f85ddf88-prpq7\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.409406 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.409325    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:14.688084 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:14.687980    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"]
Sep 08 13:55:14.691982 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:55:14.691957    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dca1633_3fb5_4cab_8d70_e996118b2f33.slice/crio-493196a52b388439914faa8f6646b3ed43ef925f37f4e7f72b0fc6a91701fe39 WatchSource:0}: Error finding container 493196a52b388439914faa8f6646b3ed43ef925f37f4e7f72b0fc6a91701fe39: Status 404 returned error can't find the container with id 493196a52b388439914faa8f6646b3ed43ef925f37f4e7f72b0fc6a91701fe39
Sep 08 13:55:15.177789 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:15.177757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" event={"ID":"4dca1633-3fb5-4cab-8d70-e996118b2f33","Type":"ContainerStarted","Data":"9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238"}
Sep 08 13:55:15.178202 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:15.177816    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" event={"ID":"4dca1633-3fb5-4cab-8d70-e996118b2f33","Type":"ContainerStarted","Data":"493196a52b388439914faa8f6646b3ed43ef925f37f4e7f72b0fc6a91701fe39"}
Sep 08 13:55:19.196749 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:19.196711    2583 generic.go:356] "Generic (PLEG): container finished" podID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerID="9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238" exitCode=0
Sep 08 13:55:19.197200 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:19.196784    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" event={"ID":"4dca1633-3fb5-4cab-8d70-e996118b2f33","Type":"ContainerDied","Data":"9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238"}
Sep 08 13:55:26.232283 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:26.232249    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" event={"ID":"4dca1633-3fb5-4cab-8d70-e996118b2f33","Type":"ContainerStarted","Data":"6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c"}
Sep 08 13:55:26.232629 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:26.232316    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" event={"ID":"4dca1633-3fb5-4cab-8d70-e996118b2f33","Type":"ContainerStarted","Data":"4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591"}
Sep 08 13:55:26.232629 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:26.232508    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:26.255824 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:26.255734    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podStartSLOduration=1.40862117 podStartE2EDuration="12.25571908s" totalImagesPullingTime="6.809431032s" totalInitContainerRuntime="4.037666878s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:55:26.254002868 +0000 UTC m=+2355.182906610" watchObservedRunningTime="2026-09-08 13:55:26.25571908 +0000 UTC m=+2355.184622822"
Sep 08 13:55:27.237067 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:27.237040    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:27.238497 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:27.238467    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 08 13:55:28.241621 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:28.241584    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 08 13:55:33.245929 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:33.245894    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:55:33.246416 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:33.246390    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 08 13:55:43.246872 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:43.246831    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 08 13:55:53.247005 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:55:53.246958    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 08 13:56:03.246664 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:03.246627    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 08 13:56:09.227283 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:09.227235    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:11.723280 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:11.723245    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:56:11.725218 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:11.725196    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 13:56:13.247343 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:13.247305    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 08 13:56:18.006146 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:18.006107    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:23.182610 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.182578    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"]
Sep 08 13:56:23.186123 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.186105    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.188525 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.188496    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 08 13:56:23.188525 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.188496    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 08 13:56:23.198872 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.198848    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"]
Sep 08 13:56:23.246950 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.246907    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 08 13:56:23.290316 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.290281    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.290498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.290337    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxn6k\" (UniqueName: \"kubernetes.io/projected/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-api-access-wxn6k\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.290498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.290450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.290498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.290493    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.290629 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.290529    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.344248 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.344211    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:23.391416 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.391386    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.391535 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.391434    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxn6k\" (UniqueName: \"kubernetes.io/projected/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-api-access-wxn6k\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.391535 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.391484    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.391535 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.391502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.391535 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.391524    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.391722 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:56:23.391636    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-serving-cert: secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 08 13:56:23.391722 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:56:23.391699    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-proxy-tls podName:a3b91e84-b083-4ddb-a3a0-74cf0d80da41 nodeName:}" failed. No retries permitted until 2026-09-08 13:56:23.891676924 +0000 UTC m=+2412.820580659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-proxy-tls") pod "isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" (UID: "a3b91e84-b083-4ddb-a3a0-74cf0d80da41") : secret "isvc-xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 08 13:56:23.391909 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.391888    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.392179 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.392161    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.392223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.392203    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.400869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.400841    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxn6k\" (UniqueName: \"kubernetes.io/projected/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-api-access-wxn6k\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.611861 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.611826    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:23.896178 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.896090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:23.898640 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:23.898610    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:24.098349 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:24.098311    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:24.374367 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:24.374339    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"]
Sep 08 13:56:24.376265 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:56:24.376241    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b91e84_b083_4ddb_a3a0_74cf0d80da41.slice/crio-38bcb1cb93061b98b8f6936c99e5e559a257f8c6e72a5e10b5a874508c7de7c6 WatchSource:0}: Error finding container 38bcb1cb93061b98b8f6936c99e5e559a257f8c6e72a5e10b5a874508c7de7c6: Status 404 returned error can't find the container with id 38bcb1cb93061b98b8f6936c99e5e559a257f8c6e72a5e10b5a874508c7de7c6
Sep 08 13:56:24.507898 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:24.507860    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" event={"ID":"a3b91e84-b083-4ddb-a3a0-74cf0d80da41","Type":"ContainerStarted","Data":"e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796"}
Sep 08 13:56:24.508139 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:24.507932    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" event={"ID":"a3b91e84-b083-4ddb-a3a0-74cf0d80da41","Type":"ContainerStarted","Data":"38bcb1cb93061b98b8f6936c99e5e559a257f8c6e72a5e10b5a874508c7de7c6"}
Sep 08 13:56:28.527248 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:28.527214    2583 generic.go:356] "Generic (PLEG): container finished" podID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerID="e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796" exitCode=0
Sep 08 13:56:28.527617 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:28.527258    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" event={"ID":"a3b91e84-b083-4ddb-a3a0-74cf0d80da41","Type":"ContainerDied","Data":"e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796"}
Sep 08 13:56:29.533801 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:29.533766    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" event={"ID":"a3b91e84-b083-4ddb-a3a0-74cf0d80da41","Type":"ContainerStarted","Data":"b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96"}
Sep 08 13:56:29.534259 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:29.533827    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" event={"ID":"a3b91e84-b083-4ddb-a3a0-74cf0d80da41","Type":"ContainerStarted","Data":"26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743"}
Sep 08 13:56:29.534259 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:29.534030    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:29.534259 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:29.534074    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:29.556006 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:29.555937    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" podStartSLOduration=2.664275988 podStartE2EDuration="6.555921012s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.891645024s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:56:29.554197921 +0000 UTC m=+2418.483101717" watchObservedRunningTime="2026-09-08 13:56:29.555921012 +0000 UTC m=+2418.484824755"
Sep 08 13:56:33.247192 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:33.247158    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 08 13:56:35.546494 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:35.546457    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:56:43.247034 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:43.246995    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:56:44.141341 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:44.141310    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"]
Sep 08 13:56:44.141589 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:44.141540    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container" containerID="cri-o://4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591" gracePeriod=30
Sep 08 13:56:44.141683 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:44.141566    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kube-rbac-proxy" containerID="cri-o://6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c" gracePeriod=30
Sep 08 13:56:44.166845 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:44.166818    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:44.605556 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:44.605527    2583 generic.go:356] "Generic (PLEG): container finished" podID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerID="6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c" exitCode=2
Sep 08 13:56:44.605901 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:44.605567    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" event={"ID":"4dca1633-3fb5-4cab-8d70-e996118b2f33","Type":"ContainerDied","Data":"6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c"}
Sep 08 13:56:48.242030 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:48.241981    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 08 13:56:49.296662 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.296627    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:49.328496 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.328462    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"]
Sep 08 13:56:49.332158 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.332136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.334150 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.334126    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 08 13:56:49.334651 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.334634    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 08 13:56:49.348068 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.348042    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"]
Sep 08 13:56:49.396461 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.396433    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:56:49.415719 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.415685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fsds\" (UniqueName: \"kubernetes.io/projected/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-api-access-5fsds\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.415864 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.415755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.415864 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.415804    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1527566-b5f3-4f1e-b076-c392cadf0ba5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.415864 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.415832    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.416087 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.415873    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.516814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.516770    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.517088 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.517058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1527566-b5f3-4f1e-b076-c392cadf0ba5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.517207 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.517112    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.517394 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.517290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.518605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.518580    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.518746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.517937    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.518943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.517572    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.518943 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.518592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsds\" (UniqueName: \"kubernetes.io/projected/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-api-access-5fsds\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.520145 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.520118    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1527566-b5f3-4f1e-b076-c392cadf0ba5-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.528674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.528624    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fsds\" (UniqueName: \"kubernetes.io/projected/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-api-access-5fsds\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.581221 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.581195    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:56:49.631217 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.631184    2583 generic.go:356] "Generic (PLEG): container finished" podID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerID="4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591" exitCode=0
Sep 08 13:56:49.631382 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.631226    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" event={"ID":"4dca1633-3fb5-4cab-8d70-e996118b2f33","Type":"ContainerDied","Data":"4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591"}
Sep 08 13:56:49.631382 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.631241    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"
Sep 08 13:56:49.631382 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.631272    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7" event={"ID":"4dca1633-3fb5-4cab-8d70-e996118b2f33","Type":"ContainerDied","Data":"493196a52b388439914faa8f6646b3ed43ef925f37f4e7f72b0fc6a91701fe39"}
Sep 08 13:56:49.631382 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.631297    2583 scope.go:117] "RemoveContainer" containerID="6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c"
Sep 08 13:56:49.639319 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.639300    2583 scope.go:117] "RemoveContainer" containerID="4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591"
Sep 08 13:56:49.642345 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.642324    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:49.647130 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.647108    2583 scope.go:117] "RemoveContainer" containerID="9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238"
Sep 08 13:56:49.654524 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.654508    2583 scope.go:117] "RemoveContainer" containerID="6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c"
Sep 08 13:56:49.654799 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:56:49.654779    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c\": container with ID starting with 6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c not found: ID does not exist" containerID="6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c"
Sep 08 13:56:49.654857 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.654808    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c"} err="failed to get container status \"6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c\": rpc error: code = NotFound desc = could not find container \"6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c\": container with ID starting with 6c957f82a6299433501fd0ddbc44006cadfbbf7badcd3f240014e6a0421cf69c not found: ID does not exist"
Sep 08 13:56:49.654857 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.654828    2583 scope.go:117] "RemoveContainer" containerID="4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591"
Sep 08 13:56:49.655064 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:56:49.655045    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591\": container with ID starting with 4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591 not found: ID does not exist" containerID="4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591"
Sep 08 13:56:49.655136 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.655076    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591"} err="failed to get container status \"4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591\": rpc error: code = NotFound desc = could not find container \"4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591\": container with ID starting with 4cad3e4a0e7845c8b5830ac5e3e1c074d18cbba34268f4dbb9ed62e26c5b8591 not found: ID does not exist"
Sep 08 13:56:49.655136 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.655101    2583 scope.go:117] "RemoveContainer" containerID="9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238"
Sep 08 13:56:49.655369 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:56:49.655354    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238\": container with ID starting with 9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238 not found: ID does not exist" containerID="9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238"
Sep 08 13:56:49.655422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.655375    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238"} err="failed to get container status \"9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238\": rpc error: code = NotFound desc = could not find container \"9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238\": container with ID starting with 9365da4a6fb9cf491c2d4887dd225036145f6b52f9903dee4c06e2821e2c3238 not found: ID does not exist"
Sep 08 13:56:49.720462 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.720433    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-cabundle-cert\") pod \"4dca1633-3fb5-4cab-8d70-e996118b2f33\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") "
Sep 08 13:56:49.720631 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.720503    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-rbac-proxy-sar-config\") pod \"4dca1633-3fb5-4cab-8d70-e996118b2f33\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") "
Sep 08 13:56:49.720703 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.720640    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dca1633-3fb5-4cab-8d70-e996118b2f33-kserve-provision-location\") pod \"4dca1633-3fb5-4cab-8d70-e996118b2f33\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") "
Sep 08 13:56:49.720703 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.720685    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nxfz\" (UniqueName: \"kubernetes.io/projected/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-api-access-5nxfz\") pod \"4dca1633-3fb5-4cab-8d70-e996118b2f33\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") "
Sep 08 13:56:49.720833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.720710    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dca1633-3fb5-4cab-8d70-e996118b2f33-proxy-tls\") pod \"4dca1633-3fb5-4cab-8d70-e996118b2f33\" (UID: \"4dca1633-3fb5-4cab-8d70-e996118b2f33\") "
Sep 08 13:56:49.720942 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.720918    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4dca1633-3fb5-4cab-8d70-e996118b2f33" (UID: "4dca1633-3fb5-4cab-8d70-e996118b2f33"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:56:49.721001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.720970    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dca1633-3fb5-4cab-8d70-e996118b2f33-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4dca1633-3fb5-4cab-8d70-e996118b2f33" (UID: "4dca1633-3fb5-4cab-8d70-e996118b2f33"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:56:49.721083 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.721055    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:56:49.721127 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.721095    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4dca1633-3fb5-4cab-8d70-e996118b2f33" (UID: "4dca1633-3fb5-4cab-8d70-e996118b2f33"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:56:49.722793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.722770    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-api-access-5nxfz" (OuterVolumeSpecName: "kube-api-access-5nxfz") pod "4dca1633-3fb5-4cab-8d70-e996118b2f33" (UID: "4dca1633-3fb5-4cab-8d70-e996118b2f33"). InnerVolumeSpecName "kube-api-access-5nxfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:56:49.722976 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.722956    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dca1633-3fb5-4cab-8d70-e996118b2f33-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4dca1633-3fb5-4cab-8d70-e996118b2f33" (UID: "4dca1633-3fb5-4cab-8d70-e996118b2f33"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:56:49.822519 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.822487    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4dca1633-3fb5-4cab-8d70-e996118b2f33-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:56:49.822519 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.822516    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nxfz\" (UniqueName: \"kubernetes.io/projected/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-api-access-5nxfz\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:56:49.822726 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.822526    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4dca1633-3fb5-4cab-8d70-e996118b2f33-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:56:49.822726 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.822536    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4dca1633-3fb5-4cab-8d70-e996118b2f33-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:56:49.917881 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.917854    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"]
Sep 08 13:56:49.920136 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:56:49.920112    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1527566_b5f3_4f1e_b076_c392cadf0ba5.slice/crio-8b4e83df67813d72c27ad5c94acb70ecd8eca8337bb0647b6c43b4b5fcd97cb6 WatchSource:0}: Error finding container 8b4e83df67813d72c27ad5c94acb70ecd8eca8337bb0647b6c43b4b5fcd97cb6: Status 404 returned error can't find the container with id 8b4e83df67813d72c27ad5c94acb70ecd8eca8337bb0647b6c43b4b5fcd97cb6
Sep 08 13:56:49.958444 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.958413    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"]
Sep 08 13:56:49.964164 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:49.964141    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-prpq7"]
Sep 08 13:56:50.638373 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:50.638339    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" event={"ID":"e1527566-b5f3-4f1e-b076-c392cadf0ba5","Type":"ContainerStarted","Data":"b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d"}
Sep 08 13:56:50.638755 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:50.638396    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" event={"ID":"e1527566-b5f3-4f1e-b076-c392cadf0ba5","Type":"ContainerStarted","Data":"8b4e83df67813d72c27ad5c94acb70ecd8eca8337bb0647b6c43b4b5fcd97cb6"}
Sep 08 13:56:51.596738 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:51.596705    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" path="/var/lib/kubelet/pods/4dca1633-3fb5-4cab-8d70-e996118b2f33/volumes"
Sep 08 13:56:54.657377 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:54.657344    2583 generic.go:356] "Generic (PLEG): container finished" podID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerID="b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d" exitCode=0
Sep 08 13:56:54.657773 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:54.657394    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" event={"ID":"e1527566-b5f3-4f1e-b076-c392cadf0ba5","Type":"ContainerDied","Data":"b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d"}
Sep 08 13:56:55.668989 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:55.668954    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" event={"ID":"e1527566-b5f3-4f1e-b076-c392cadf0ba5","Type":"ContainerStarted","Data":"053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb"}
Sep 08 13:56:55.669462 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:55.669073    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" event={"ID":"e1527566-b5f3-4f1e-b076-c392cadf0ba5","Type":"ContainerStarted","Data":"960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98"}
Sep 08 13:56:55.669462 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:55.669391    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:55.669462 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:55.669446    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:56:55.670784 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:55.670762    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:56:55.692687 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:55.692631    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podStartSLOduration=2.908540207 podStartE2EDuration="6.692614983s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.784074776s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:56:55.691546641 +0000 UTC m=+2444.620450381" watchObservedRunningTime="2026-09-08 13:56:55.692614983 +0000 UTC m=+2444.621518724"
Sep 08 13:56:56.673033 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:56:56.672992    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:57:01.677336 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:01.677308    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:57:01.677833 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:01.677807    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:57:05.550444 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:05.550412    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:57:11.677866 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:11.677821    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:57:13.271767 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:13.271728    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"]
Sep 08 13:57:13.272256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:13.272053    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="kserve-container" containerID="cri-o://26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743" gracePeriod=30
Sep 08 13:57:13.272256 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:13.272067    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="kube-rbac-proxy" containerID="cri-o://b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96" gracePeriod=30
Sep 08 13:57:13.305469 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:13.305430    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:13.751828 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:13.751792    2583 generic.go:356] "Generic (PLEG): container finished" podID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerID="b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96" exitCode=2
Sep 08 13:57:13.752042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:13.751866    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" event={"ID":"a3b91e84-b083-4ddb-a3a0-74cf0d80da41","Type":"ContainerDied","Data":"b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96"}
Sep 08 13:57:15.542069 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:15.541993    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 08 13:57:18.447885 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.447847    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:18.481923 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.481886    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"]
Sep 08 13:57:18.482326 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.482308    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container"
Sep 08 13:57:18.482422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.482328    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container"
Sep 08 13:57:18.482422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.482361    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kube-rbac-proxy"
Sep 08 13:57:18.482422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.482369    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kube-rbac-proxy"
Sep 08 13:57:18.482422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.482393    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="storage-initializer"
Sep 08 13:57:18.482422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.482401    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="storage-initializer"
Sep 08 13:57:18.482680 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.482503    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kserve-container"
Sep 08 13:57:18.482680 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.482518    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dca1633-3fb5-4cab-8d70-e996118b2f33" containerName="kube-rbac-proxy"
Sep 08 13:57:18.485425 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.485405    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.487660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.487639    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 08 13:57:18.487882 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.487866    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 08 13:57:18.501634 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.501604    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"]
Sep 08 13:57:18.581899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.581859    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.582113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.581924    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxvwp\" (UniqueName: \"kubernetes.io/projected/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-api-access-vxvwp\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.582113 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.581993    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.582243 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.582111    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.582243 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.582161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.683075 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.683040    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.683264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.683083    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxvwp\" (UniqueName: \"kubernetes.io/projected/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-api-access-vxvwp\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.683264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.683114    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.683264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.683169    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.683264 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.683211    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.683264 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:57:18.683221    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 08 13:57:18.683541 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:57:18.683294    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-proxy-tls podName:7fa3a82f-29d2-4d18-969e-ccfe7e2e52da nodeName:}" failed. No retries permitted until 2026-09-08 13:57:19.183274351 +0000 UTC m=+2468.112178076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" (UID: "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 08 13:57:18.683665 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.683644    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.683873 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.683851    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.683873 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.683868    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:18.697976 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:18.697918    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxvwp\" (UniqueName: \"kubernetes.io/projected/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-api-access-vxvwp\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:19.119839 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:19.119772    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:57:19.189771 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:19.189737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:19.192490 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:19.192455    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:19.397398 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:19.397293    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:19.677588 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:19.677508    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"]
Sep 08 13:57:19.680307 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:57:19.680279    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa3a82f_29d2_4d18_969e_ccfe7e2e52da.slice/crio-2421ce1ad0272f85196b13d3945e44802d4abebf7494534bd26259f075978501 WatchSource:0}: Error finding container 2421ce1ad0272f85196b13d3945e44802d4abebf7494534bd26259f075978501: Status 404 returned error can't find the container with id 2421ce1ad0272f85196b13d3945e44802d4abebf7494534bd26259f075978501
Sep 08 13:57:19.780101 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:19.780065    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" event={"ID":"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da","Type":"ContainerStarted","Data":"09c1c3cc73646b23b20f080ff3ca82e072c55dca2b8a914545399e6fd340dc76"}
Sep 08 13:57:19.780263 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:19.780125    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" event={"ID":"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da","Type":"ContainerStarted","Data":"2421ce1ad0272f85196b13d3945e44802d4abebf7494534bd26259f075978501"}
Sep 08 13:57:20.028714 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.028687    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:57:20.098614 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.098573    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-cabundle-cert\") pod \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") "
Sep 08 13:57:20.098796 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.098638    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-rbac-proxy-sar-config\") pod \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") "
Sep 08 13:57:20.098796 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.098656    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-proxy-tls\") pod \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") "
Sep 08 13:57:20.098796 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.098708    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kserve-provision-location\") pod \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") "
Sep 08 13:57:20.098796 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.098724    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxn6k\" (UniqueName: \"kubernetes.io/projected/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-api-access-wxn6k\") pod \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\" (UID: \"a3b91e84-b083-4ddb-a3a0-74cf0d80da41\") "
Sep 08 13:57:20.099063 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.098953    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3b91e84-b083-4ddb-a3a0-74cf0d80da41" (UID: "a3b91e84-b083-4ddb-a3a0-74cf0d80da41"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:57:20.099063 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.098974    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3b91e84-b083-4ddb-a3a0-74cf0d80da41" (UID: "a3b91e84-b083-4ddb-a3a0-74cf0d80da41"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:57:20.099162 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.099058    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3b91e84-b083-4ddb-a3a0-74cf0d80da41" (UID: "a3b91e84-b083-4ddb-a3a0-74cf0d80da41"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:57:20.100990 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.100968    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-api-access-wxn6k" (OuterVolumeSpecName: "kube-api-access-wxn6k") pod "a3b91e84-b083-4ddb-a3a0-74cf0d80da41" (UID: "a3b91e84-b083-4ddb-a3a0-74cf0d80da41"). InnerVolumeSpecName "kube-api-access-wxn6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:57:20.101102 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.101076    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3b91e84-b083-4ddb-a3a0-74cf0d80da41" (UID: "a3b91e84-b083-4ddb-a3a0-74cf0d80da41"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:57:20.199778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.199689    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:57:20.199778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.199722    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:57:20.199778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.199732    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:57:20.199778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.199742    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:57:20.199778 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.199751    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxn6k\" (UniqueName: \"kubernetes.io/projected/a3b91e84-b083-4ddb-a3a0-74cf0d80da41-kube-api-access-wxn6k\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:57:20.786698 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.786657    2583 generic.go:356] "Generic (PLEG): container finished" podID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerID="26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743" exitCode=0
Sep 08 13:57:20.787223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.786718    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"
Sep 08 13:57:20.787223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.786748    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" event={"ID":"a3b91e84-b083-4ddb-a3a0-74cf0d80da41","Type":"ContainerDied","Data":"26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743"}
Sep 08 13:57:20.787223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.786808    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk" event={"ID":"a3b91e84-b083-4ddb-a3a0-74cf0d80da41","Type":"ContainerDied","Data":"38bcb1cb93061b98b8f6936c99e5e559a257f8c6e72a5e10b5a874508c7de7c6"}
Sep 08 13:57:20.787223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.786834    2583 scope.go:117] "RemoveContainer" containerID="b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96"
Sep 08 13:57:20.795822 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.795808    2583 scope.go:117] "RemoveContainer" containerID="26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743"
Sep 08 13:57:20.803499 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.803476    2583 scope.go:117] "RemoveContainer" containerID="e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796"
Sep 08 13:57:20.809310 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.809287    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"]
Sep 08 13:57:20.811272 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.811254    2583 scope.go:117] "RemoveContainer" containerID="b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96"
Sep 08 13:57:20.811596 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:57:20.811568    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96\": container with ID starting with b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96 not found: ID does not exist" containerID="b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96"
Sep 08 13:57:20.811697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.811603    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96"} err="failed to get container status \"b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96\": rpc error: code = NotFound desc = could not find container \"b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96\": container with ID starting with b2e78827b894341cb0584df7c8bea15ce20f2a29ad7293738a19a417b505fe96 not found: ID does not exist"
Sep 08 13:57:20.811697 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.811628    2583 scope.go:117] "RemoveContainer" containerID="26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743"
Sep 08 13:57:20.812101 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:57:20.811980    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743\": container with ID starting with 26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743 not found: ID does not exist" containerID="26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743"
Sep 08 13:57:20.812101 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.812059    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743"} err="failed to get container status \"26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743\": rpc error: code = NotFound desc = could not find container \"26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743\": container with ID starting with 26859ee33c5c76a2bd687fc92990dcc63d7c03914eb0f43941f4ca1ee9272743 not found: ID does not exist"
Sep 08 13:57:20.812101 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.812088    2583 scope.go:117] "RemoveContainer" containerID="e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796"
Sep 08 13:57:20.812476 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:57:20.812448    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796\": container with ID starting with e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796 not found: ID does not exist" containerID="e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796"
Sep 08 13:57:20.812571 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.812484    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796"} err="failed to get container status \"e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796\": rpc error: code = NotFound desc = could not find container \"e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796\": container with ID starting with e0203d27b9e15d743071345d071e77508460cf88046c836d3516ae0858d85796 not found: ID does not exist"
Sep 08 13:57:20.814193 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:20.814175    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wj9sk"]
Sep 08 13:57:21.595107 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:21.595068    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" path="/var/lib/kubelet/pods/a3b91e84-b083-4ddb-a3a0-74cf0d80da41/volumes"
Sep 08 13:57:21.678535 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:21.678496    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:57:23.803180 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:23.803095    2583 generic.go:356] "Generic (PLEG): container finished" podID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerID="09c1c3cc73646b23b20f080ff3ca82e072c55dca2b8a914545399e6fd340dc76" exitCode=0
Sep 08 13:57:23.803180 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:23.803169    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" event={"ID":"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da","Type":"ContainerDied","Data":"09c1c3cc73646b23b20f080ff3ca82e072c55dca2b8a914545399e6fd340dc76"}
Sep 08 13:57:24.811203 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:24.811159    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" event={"ID":"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da","Type":"ContainerStarted","Data":"d11e75d8fcffddffd6c8bc4c683479dad4a194a82f7a292aad2e07d51626a9eb"}
Sep 08 13:57:24.811866 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:24.811242    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" event={"ID":"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da","Type":"ContainerStarted","Data":"8daef5cd27114bf52d0abd689093129a6ab25a2ac07991af0307a2f18d9a21a9"}
Sep 08 13:57:24.811866 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:24.811488    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:24.840492 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:24.840436    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" podStartSLOduration=3.143580641 podStartE2EDuration="6.840417798s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.696837157s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:57:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:57:24.839789368 +0000 UTC m=+2473.768693132" watchObservedRunningTime="2026-09-08 13:57:24.840417798 +0000 UTC m=+2473.769321539"
Sep 08 13:57:25.816574 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:25.816544    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:31.678704 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:31.678668    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:57:31.833313 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:31.833285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:57:41.678047 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:41.677985    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:57:51.678040 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:57:51.677984    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:58:01.678322 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:01.678280    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:58:01.837625 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:01.837587    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:58:08.515817 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:08.515722    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"]
Sep 08 13:58:08.516422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:08.515971    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kserve-container" containerID="cri-o://8daef5cd27114bf52d0abd689093129a6ab25a2ac07991af0307a2f18d9a21a9" gracePeriod=30
Sep 08 13:58:08.516422 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:08.516200    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kube-rbac-proxy" containerID="cri-o://d11e75d8fcffddffd6c8bc4c683479dad4a194a82f7a292aad2e07d51626a9eb" gracePeriod=30
Sep 08 13:58:08.589548 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:08.589508    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:09.021821 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:09.021789    2583 generic.go:356] "Generic (PLEG): container finished" podID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerID="d11e75d8fcffddffd6c8bc4c683479dad4a194a82f7a292aad2e07d51626a9eb" exitCode=2
Sep 08 13:58:09.021992 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:09.021845    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" event={"ID":"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da","Type":"ContainerDied","Data":"d11e75d8fcffddffd6c8bc4c683479dad4a194a82f7a292aad2e07d51626a9eb"}
Sep 08 13:58:11.678518 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:11.678488    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:58:11.828848 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:11.828801    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 08 13:58:11.834673 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:11.834640    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.47:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 08 13:58:13.674050 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.673990    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:13.721653 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.721609    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"]
Sep 08 13:58:13.722105 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.722086    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="storage-initializer"
Sep 08 13:58:13.722204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.722106    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="storage-initializer"
Sep 08 13:58:13.722204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.722136    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="kserve-container"
Sep 08 13:58:13.722204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.722144    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="kserve-container"
Sep 08 13:58:13.722204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.722168    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="kube-rbac-proxy"
Sep 08 13:58:13.722204 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.722176    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="kube-rbac-proxy"
Sep 08 13:58:13.722463 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.722262    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="kserve-container"
Sep 08 13:58:13.722463 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.722276    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3b91e84-b083-4ddb-a3a0-74cf0d80da41" containerName="kube-rbac-proxy"
Sep 08 13:58:13.726764 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.726741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.729216 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.729185    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 08 13:58:13.729331 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.729260    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 08 13:58:13.738258 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.738232    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"]
Sep 08 13:58:13.866944 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.866912    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.867140 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.866959    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-api-access-gn9cw\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.867140 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.866978    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.867237 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.867155    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a7309e-d080-435f-98a1-ab951b1e9a74-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.867237 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.867196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a7309e-d080-435f-98a1-ab951b1e9a74-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.968217 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.968133    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-api-access-gn9cw\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.968217 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.968176    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.968457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.968247    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a7309e-d080-435f-98a1-ab951b1e9a74-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.968457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.968279    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a7309e-d080-435f-98a1-ab951b1e9a74-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.968457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.968312    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.968842 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.968815    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a7309e-d080-435f-98a1-ab951b1e9a74-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.969131 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.969106    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.969638 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.969610    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:13.969747 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.969666    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.971069 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.971048    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a7309e-d080-435f-98a1-ab951b1e9a74-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:13.978202 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:13.978185    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-api-access-gn9cw\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-2ml6p\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:14.039201 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:14.039167    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:14.318341 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:14.318180    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"]
Sep 08 13:58:15.050392 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.050357    2583 generic.go:356] "Generic (PLEG): container finished" podID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerID="8daef5cd27114bf52d0abd689093129a6ab25a2ac07991af0307a2f18d9a21a9" exitCode=0
Sep 08 13:58:15.050814 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.050410    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" event={"ID":"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da","Type":"ContainerDied","Data":"8daef5cd27114bf52d0abd689093129a6ab25a2ac07991af0307a2f18d9a21a9"}
Sep 08 13:58:15.052091 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.052067    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" event={"ID":"26a7309e-d080-435f-98a1-ab951b1e9a74","Type":"ContainerStarted","Data":"d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4"}
Sep 08 13:58:15.052216 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.052120    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" event={"ID":"26a7309e-d080-435f-98a1-ab951b1e9a74","Type":"ContainerStarted","Data":"0e8b503bb6014f1942c80af7a7536b984abc85db6894d25fb10e8dc54620804f"}
Sep 08 13:58:15.080493 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.080454    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:58:15.179133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179056    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-proxy-tls\") pod \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") "
Sep 08 13:58:15.179133 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179098    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxvwp\" (UniqueName: \"kubernetes.io/projected/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-api-access-vxvwp\") pod \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") "
Sep 08 13:58:15.179372 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179194    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kserve-provision-location\") pod \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") "
Sep 08 13:58:15.179372 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179264    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-cabundle-cert\") pod \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") "
Sep 08 13:58:15.179372 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179293    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-rbac-proxy-sar-config\") pod \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\" (UID: \"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da\") "
Sep 08 13:58:15.179538 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179511    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" (UID: "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:15.179624 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179597    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" (UID: "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:58:15.179683 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179648    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" (UID: "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:58:15.179881 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179865    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:15.179977 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179886    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:15.179977 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.179900    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:15.181451 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.181421    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" (UID: "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:58:15.181566 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.181550    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-api-access-vxvwp" (OuterVolumeSpecName: "kube-api-access-vxvwp") pod "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" (UID: "7fa3a82f-29d2-4d18-969e-ccfe7e2e52da"). InnerVolumeSpecName "kube-api-access-vxvwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:58:15.280608 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.280563    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:15.280608 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:15.280599    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxvwp\" (UniqueName: \"kubernetes.io/projected/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da-kube-api-access-vxvwp\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:16.058788 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:16.058713    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"
Sep 08 13:58:16.058788 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:16.058733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9" event={"ID":"7fa3a82f-29d2-4d18-969e-ccfe7e2e52da","Type":"ContainerDied","Data":"2421ce1ad0272f85196b13d3945e44802d4abebf7494534bd26259f075978501"}
Sep 08 13:58:16.059319 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:16.058797    2583 scope.go:117] "RemoveContainer" containerID="d11e75d8fcffddffd6c8bc4c683479dad4a194a82f7a292aad2e07d51626a9eb"
Sep 08 13:58:16.067605 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:16.067433    2583 scope.go:117] "RemoveContainer" containerID="8daef5cd27114bf52d0abd689093129a6ab25a2ac07991af0307a2f18d9a21a9"
Sep 08 13:58:16.075313 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:16.075295    2583 scope.go:117] "RemoveContainer" containerID="09c1c3cc73646b23b20f080ff3ca82e072c55dca2b8a914545399e6fd340dc76"
Sep 08 13:58:16.079518 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:16.079494    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"]
Sep 08 13:58:16.087188 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:16.087161    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-r2wr9"]
Sep 08 13:58:17.595055 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:17.594998    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" path="/var/lib/kubelet/pods/7fa3a82f-29d2-4d18-969e-ccfe7e2e52da/volumes"
Sep 08 13:58:19.079141 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:19.079103    2583 generic.go:356] "Generic (PLEG): container finished" podID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerID="d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4" exitCode=0
Sep 08 13:58:19.079535 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:19.079179    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" event={"ID":"26a7309e-d080-435f-98a1-ab951b1e9a74","Type":"ContainerDied","Data":"d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4"}
Sep 08 13:58:19.080570 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:19.080553    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 13:58:19.840389 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:19.840353    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"]
Sep 08 13:58:19.840621 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:19.840584    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" containerID="cri-o://960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98" gracePeriod=30
Sep 08 13:58:19.840688 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:19.840624    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kube-rbac-proxy" containerID="cri-o://053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb" gracePeriod=30
Sep 08 13:58:19.877371 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:19.877334    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:20.088522 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:20.088483    2583 generic.go:356] "Generic (PLEG): container finished" podID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerID="053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb" exitCode=2
Sep 08 13:58:20.088899 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:20.088552    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" event={"ID":"e1527566-b5f3-4f1e-b076-c392cadf0ba5","Type":"ContainerDied","Data":"053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb"}
Sep 08 13:58:21.673457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:21.673414    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 08 13:58:21.678787 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:21.678755    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 08 13:58:24.913436 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.913354    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:24.945143 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.945107    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"]
Sep 08 13:58:24.945646 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.945623    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kserve-container"
Sep 08 13:58:24.945646 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.945643    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kserve-container"
Sep 08 13:58:24.945832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.945660    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="storage-initializer"
Sep 08 13:58:24.945832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.945669    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="storage-initializer"
Sep 08 13:58:24.945832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.945700    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kube-rbac-proxy"
Sep 08 13:58:24.945832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.945710    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kube-rbac-proxy"
Sep 08 13:58:24.945832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.945788    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kube-rbac-proxy"
Sep 08 13:58:24.945832 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.945801    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fa3a82f-29d2-4d18-969e-ccfe7e2e52da" containerName="kserve-container"
Sep 08 13:58:24.949563 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.949539    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:24.951803 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.951779    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 08 13:58:24.952117 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.952095    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 08 13:58:24.960547 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:24.960521    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"]
Sep 08 13:58:25.076242 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.076208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.076440 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.076262    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb6a516-fe47-4857-90ed-1ff5bab7484a-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.076440 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.076340    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.076564 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.076450    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.076564 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.076494    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-api-access-cls4g\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.177636 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.177544    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.177636 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.177603    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-api-access-cls4g\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.177897 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.177653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.177897 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.177691    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb6a516-fe47-4857-90ed-1ff5bab7484a-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.177897 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.177727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.178109 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:58:25.178064    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 08 13:58:25.178168 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.178120    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.178168 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:58:25.178129    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecb6a516-fe47-4857-90ed-1ff5bab7484a-proxy-tls podName:ecb6a516-fe47-4857-90ed-1ff5bab7484a nodeName:}" failed. No retries permitted until 2026-09-08 13:58:25.678107202 +0000 UTC m=+2534.607010922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ecb6a516-fe47-4857-90ed-1ff5bab7484a-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" (UID: "ecb6a516-fe47-4857-90ed-1ff5bab7484a") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 08 13:58:25.178343 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.178318    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.178501 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.178480    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.188886 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.188862    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-api-access-cls4g\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.382192 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.382154    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:58:25.682049 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.681990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb6a516-fe47-4857-90ed-1ff5bab7484a-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.684978 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.684950    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb6a516-fe47-4857-90ed-1ff5bab7484a-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.863866 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.863824    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:25.909402 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:25.909358    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:58:26.085947 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.085904    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-cabundle-cert\") pod \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") "
Sep 08 13:58:26.086457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.085962    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kserve-provision-location\") pod \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") "
Sep 08 13:58:26.086457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.086029    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fsds\" (UniqueName: \"kubernetes.io/projected/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-api-access-5fsds\") pod \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") "
Sep 08 13:58:26.086457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.086129    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-rbac-proxy-sar-config\") pod \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") "
Sep 08 13:58:26.086457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.086166    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1527566-b5f3-4f1e-b076-c392cadf0ba5-proxy-tls\") pod \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\" (UID: \"e1527566-b5f3-4f1e-b076-c392cadf0ba5\") "
Sep 08 13:58:26.086457 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.086370    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e1527566-b5f3-4f1e-b076-c392cadf0ba5" (UID: "e1527566-b5f3-4f1e-b076-c392cadf0ba5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:58:26.086739 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.086374    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e1527566-b5f3-4f1e-b076-c392cadf0ba5" (UID: "e1527566-b5f3-4f1e-b076-c392cadf0ba5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:58:26.086739 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.086540    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1527566-b5f3-4f1e-b076-c392cadf0ba5" (UID: "e1527566-b5f3-4f1e-b076-c392cadf0ba5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:58:26.088682 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.088656    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-api-access-5fsds" (OuterVolumeSpecName: "kube-api-access-5fsds") pod "e1527566-b5f3-4f1e-b076-c392cadf0ba5" (UID: "e1527566-b5f3-4f1e-b076-c392cadf0ba5"). InnerVolumeSpecName "kube-api-access-5fsds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:58:26.088804 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.088714    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1527566-b5f3-4f1e-b076-c392cadf0ba5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1527566-b5f3-4f1e-b076-c392cadf0ba5" (UID: "e1527566-b5f3-4f1e-b076-c392cadf0ba5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:58:26.125100 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.125068    2583 generic.go:356] "Generic (PLEG): container finished" podID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerID="960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98" exitCode=0
Sep 08 13:58:26.125269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.125153    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"
Sep 08 13:58:26.125269 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.125156    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" event={"ID":"e1527566-b5f3-4f1e-b076-c392cadf0ba5","Type":"ContainerDied","Data":"960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98"}
Sep 08 13:58:26.125358 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.125326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2" event={"ID":"e1527566-b5f3-4f1e-b076-c392cadf0ba5","Type":"ContainerDied","Data":"8b4e83df67813d72c27ad5c94acb70ecd8eca8337bb0647b6c43b4b5fcd97cb6"}
Sep 08 13:58:26.125412 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.125364    2583 scope.go:117] "RemoveContainer" containerID="053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb"
Sep 08 13:58:26.135415 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.135391    2583 scope.go:117] "RemoveContainer" containerID="960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98"
Sep 08 13:58:26.145707 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.145671    2583 scope.go:117] "RemoveContainer" containerID="b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d"
Sep 08 13:58:26.152998 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.152975    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"]
Sep 08 13:58:26.157141 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.156769    2583 scope.go:117] "RemoveContainer" containerID="053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb"
Sep 08 13:58:26.157141 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.156890    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"]
Sep 08 13:58:26.157754 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:58:26.157480    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb\": container with ID starting with 053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb not found: ID does not exist" containerID="053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb"
Sep 08 13:58:26.157754 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.157514    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb"} err="failed to get container status \"053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb\": rpc error: code = NotFound desc = could not find container \"053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb\": container with ID starting with 053ce173c8975b9e134289cbe3f58c986f40a09287b252e550ce3e400246fbdb not found: ID does not exist"
Sep 08 13:58:26.157754 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.157534    2583 scope.go:117] "RemoveContainer" containerID="960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98"
Sep 08 13:58:26.157968 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:58:26.157948    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98\": container with ID starting with 960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98 not found: ID does not exist" containerID="960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98"
Sep 08 13:58:26.158069 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.157977    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98"} err="failed to get container status \"960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98\": rpc error: code = NotFound desc = could not find container \"960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98\": container with ID starting with 960e9e6ea7fde66e3f2a36da12a6a75d6948a3b02cd109a7e6f0b18154b4dc98 not found: ID does not exist"
Sep 08 13:58:26.158069 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.157999    2583 scope.go:117] "RemoveContainer" containerID="b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d"
Sep 08 13:58:26.158476 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:58:26.158433    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d\": container with ID starting with b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d not found: ID does not exist" containerID="b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d"
Sep 08 13:58:26.158476 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.158460    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d"} err="failed to get container status \"b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d\": rpc error: code = NotFound desc = could not find container \"b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d\": container with ID starting with b0761d51a30ad374a66cda6d05d08c3716b0fa9c3ef55a7ca59c9c5f067cd87d not found: ID does not exist"
Sep 08 13:58:26.159478 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.159457    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-92wj2"]
Sep 08 13:58:26.159632 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:58:26.159613    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb6a516_fe47_4857_90ed_1ff5bab7484a.slice/crio-83e00116f655bc960a20ebebfc772b5120a3dafee95736f1576aa9e61c127e1a WatchSource:0}: Error finding container 83e00116f655bc960a20ebebfc772b5120a3dafee95736f1576aa9e61c127e1a: Status 404 returned error can't find the container with id 83e00116f655bc960a20ebebfc772b5120a3dafee95736f1576aa9e61c127e1a
Sep 08 13:58:26.187824 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.187796    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:26.187930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.187826    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:26.187930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.187842    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fsds\" (UniqueName: \"kubernetes.io/projected/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-api-access-5fsds\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:26.187930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.187856    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1527566-b5f3-4f1e-b076-c392cadf0ba5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:26.187930 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:26.187870    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1527566-b5f3-4f1e-b076-c392cadf0ba5-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:58:27.133785 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:27.133750    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" event={"ID":"ecb6a516-fe47-4857-90ed-1ff5bab7484a","Type":"ContainerStarted","Data":"f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3"}
Sep 08 13:58:27.134233 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:27.133815    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" event={"ID":"ecb6a516-fe47-4857-90ed-1ff5bab7484a","Type":"ContainerStarted","Data":"83e00116f655bc960a20ebebfc772b5120a3dafee95736f1576aa9e61c127e1a"}
Sep 08 13:58:27.596183 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:27.596144    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" path="/var/lib/kubelet/pods/e1527566-b5f3-4f1e-b076-c392cadf0ba5/volumes"
Sep 08 13:58:36.180800 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:36.180697    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" event={"ID":"26a7309e-d080-435f-98a1-ab951b1e9a74","Type":"ContainerStarted","Data":"924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc"}
Sep 08 13:58:36.180800 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:36.180768    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" event={"ID":"26a7309e-d080-435f-98a1-ab951b1e9a74","Type":"ContainerStarted","Data":"0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0"}
Sep 08 13:58:36.181488 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:36.180973    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:36.182283 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:36.182257    2583 generic.go:356] "Generic (PLEG): container finished" podID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerID="f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3" exitCode=0
Sep 08 13:58:36.182364 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:36.182326    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" event={"ID":"ecb6a516-fe47-4857-90ed-1ff5bab7484a","Type":"ContainerDied","Data":"f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3"}
Sep 08 13:58:36.206712 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:36.206660    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podStartSLOduration=2.654141231 podStartE2EDuration="23.206644178s" totalImagesPullingTime="16.738965746s" totalInitContainerRuntime="3.813537201s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:58:36.20462983 +0000 UTC m=+2545.133533589" watchObservedRunningTime="2026-09-08 13:58:36.206644178 +0000 UTC m=+2545.135547920"
Sep 08 13:58:37.189498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:37.189465    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" event={"ID":"ecb6a516-fe47-4857-90ed-1ff5bab7484a","Type":"ContainerStarted","Data":"d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138"}
Sep 08 13:58:37.189894 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:37.189740    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:37.189894 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:37.189796    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" event={"ID":"ecb6a516-fe47-4857-90ed-1ff5bab7484a","Type":"ContainerStarted","Data":"8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7"}
Sep 08 13:58:37.190057 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:37.189916    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:37.191373 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:37.191348    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 08 13:58:37.216042 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:37.210487    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" podStartSLOduration=9.23014281 podStartE2EDuration="13.210466062s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.980323252s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:58:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:58:37.209650546 +0000 UTC m=+2546.138554289" watchObservedRunningTime="2026-09-08 13:58:37.210466062 +0000 UTC m=+2546.139369796"
Sep 08 13:58:38.194134 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:38.194106    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:38.194529 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:38.194272    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 08 13:58:43.198189 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:43.198154    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:58:43.198692 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:43.198584    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 08 13:58:44.203330 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:44.203300    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:58:53.198643 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:58:53.198602    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 08 13:59:03.198848 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:03.198809    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 08 13:59:13.199356 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:13.199316    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 08 13:59:14.207657 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:14.207629    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:59:15.009781 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:15.009745    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"]
Sep 08 13:59:15.010106 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:15.010065    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="kserve-container" containerID="cri-o://8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7" gracePeriod=30
Sep 08 13:59:15.010254 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:15.010071    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="kube-rbac-proxy" containerID="cri-o://d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138" gracePeriod=30
Sep 08 13:59:15.044086 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:15.044053    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:15.365395 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:15.365361    2583 generic.go:356] "Generic (PLEG): container finished" podID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerID="d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138" exitCode=2
Sep 08 13:59:15.365780 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:15.365412    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" event={"ID":"ecb6a516-fe47-4857-90ed-1ff5bab7484a","Type":"ContainerDied","Data":"d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138"}
Sep 08 13:59:16.047938 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.047915    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:59:16.196519 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.196418    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb6a516-fe47-4857-90ed-1ff5bab7484a-proxy-tls\") pod \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") "
Sep 08 13:59:16.196519 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.196473    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-cabundle-cert\") pod \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") "
Sep 08 13:59:16.196519 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.196515    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-rbac-proxy-sar-config\") pod \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") "
Sep 08 13:59:16.196795 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.196549    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-api-access-cls4g\") pod \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") "
Sep 08 13:59:16.196795 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.196625    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kserve-provision-location\") pod \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\" (UID: \"ecb6a516-fe47-4857-90ed-1ff5bab7484a\") "
Sep 08 13:59:16.196907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.196872    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecb6a516-fe47-4857-90ed-1ff5bab7484a" (UID: "ecb6a516-fe47-4857-90ed-1ff5bab7484a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:59:16.196967 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.196902    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecb6a516-fe47-4857-90ed-1ff5bab7484a" (UID: "ecb6a516-fe47-4857-90ed-1ff5bab7484a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:59:16.197123 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.197102    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecb6a516-fe47-4857-90ed-1ff5bab7484a" (UID: "ecb6a516-fe47-4857-90ed-1ff5bab7484a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:16.198746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.198729    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb6a516-fe47-4857-90ed-1ff5bab7484a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecb6a516-fe47-4857-90ed-1ff5bab7484a" (UID: "ecb6a516-fe47-4857-90ed-1ff5bab7484a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:59:16.198887 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.198872    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-api-access-cls4g" (OuterVolumeSpecName: "kube-api-access-cls4g") pod "ecb6a516-fe47-4857-90ed-1ff5bab7484a" (UID: "ecb6a516-fe47-4857-90ed-1ff5bab7484a"). InnerVolumeSpecName "kube-api-access-cls4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:59:16.297707 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.297670    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:16.297707 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.297696    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cls4g\" (UniqueName: \"kubernetes.io/projected/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kube-api-access-cls4g\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:16.297707 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.297706    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb6a516-fe47-4857-90ed-1ff5bab7484a-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:16.297707 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.297715    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb6a516-fe47-4857-90ed-1ff5bab7484a-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:16.297977 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.297726    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb6a516-fe47-4857-90ed-1ff5bab7484a-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:16.372001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.371957    2583 generic.go:356] "Generic (PLEG): container finished" podID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerID="8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7" exitCode=0
Sep 08 13:59:16.372476 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.372046    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"
Sep 08 13:59:16.372476 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.372059    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" event={"ID":"ecb6a516-fe47-4857-90ed-1ff5bab7484a","Type":"ContainerDied","Data":"8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7"}
Sep 08 13:59:16.372476 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.372142    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml" event={"ID":"ecb6a516-fe47-4857-90ed-1ff5bab7484a","Type":"ContainerDied","Data":"83e00116f655bc960a20ebebfc772b5120a3dafee95736f1576aa9e61c127e1a"}
Sep 08 13:59:16.372476 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.372185    2583 scope.go:117] "RemoveContainer" containerID="d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138"
Sep 08 13:59:16.381022 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.380991    2583 scope.go:117] "RemoveContainer" containerID="8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7"
Sep 08 13:59:16.388891 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.388873    2583 scope.go:117] "RemoveContainer" containerID="f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3"
Sep 08 13:59:16.397531 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.397150    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"]
Sep 08 13:59:16.397947 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.397927    2583 scope.go:117] "RemoveContainer" containerID="d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138"
Sep 08 13:59:16.398301 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:16.398259    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138\": container with ID starting with d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138 not found: ID does not exist" containerID="d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138"
Sep 08 13:59:16.398374 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.398312    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138"} err="failed to get container status \"d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138\": rpc error: code = NotFound desc = could not find container \"d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138\": container with ID starting with d5892c70dc0be4ea5dcf19683423e59ec1659d23bed2400f2dbcd0ac1c3e3138 not found: ID does not exist"
Sep 08 13:59:16.398374 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.398332    2583 scope.go:117] "RemoveContainer" containerID="8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7"
Sep 08 13:59:16.398582 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:16.398568    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7\": container with ID starting with 8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7 not found: ID does not exist" containerID="8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7"
Sep 08 13:59:16.398630 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.398586    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7"} err="failed to get container status \"8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7\": rpc error: code = NotFound desc = could not find container \"8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7\": container with ID starting with 8b5461a64c86f88e75a4035717d8d46541242a334e3e0cd04b9d8c1bdab6dae7 not found: ID does not exist"
Sep 08 13:59:16.398630 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.398598    2583 scope.go:117] "RemoveContainer" containerID="f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3"
Sep 08 13:59:16.398892 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:16.398869    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3\": container with ID starting with f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3 not found: ID does not exist" containerID="f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3"
Sep 08 13:59:16.398949 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.398902    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3"} err="failed to get container status \"f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3\": rpc error: code = NotFound desc = could not find container \"f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3\": container with ID starting with f10938bf39c78dc52d0c29b587d0499620f7204baf6b8b56fc4349f5c5b57ef3 not found: ID does not exist"
Sep 08 13:59:16.399286 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:16.399268    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-gsdml"]
Sep 08 13:59:17.594953 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:17.594919    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" path="/var/lib/kubelet/pods/ecb6a516-fe47-4857-90ed-1ff5bab7484a/volumes"
Sep 08 13:59:20.135314 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.135275    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:20.170184 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170152    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"]
Sep 08 13:59:20.170498 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170483    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kube-rbac-proxy"
Sep 08 13:59:20.170557 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170503    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kube-rbac-proxy"
Sep 08 13:59:20.170557 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170523    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container"
Sep 08 13:59:20.170557 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170530    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container"
Sep 08 13:59:20.170557 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170537    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="storage-initializer"
Sep 08 13:59:20.170557 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170543    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="storage-initializer"
Sep 08 13:59:20.170557 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170556    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="kube-rbac-proxy"
Sep 08 13:59:20.170746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170565    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="kube-rbac-proxy"
Sep 08 13:59:20.170746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170574    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="kserve-container"
Sep 08 13:59:20.170746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170580    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="kserve-container"
Sep 08 13:59:20.170746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170594    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="storage-initializer"
Sep 08 13:59:20.170746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170599    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="storage-initializer"
Sep 08 13:59:20.170746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170649    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kserve-container"
Sep 08 13:59:20.170746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170656    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1527566-b5f3-4f1e-b076-c392cadf0ba5" containerName="kube-rbac-proxy"
Sep 08 13:59:20.170746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170663    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="kube-rbac-proxy"
Sep 08 13:59:20.170746 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.170670    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb6a516-fe47-4857-90ed-1ff5bab7484a" containerName="kserve-container"
Sep 08 13:59:20.173065 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.173038    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.174967 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.174944    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 08 13:59:20.175188 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.175174    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 08 13:59:20.183927 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.183907    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"]
Sep 08 13:59:20.246535 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.246499    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:20.336663 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.336622    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9611d1df-c911-4812-8910-e3e530ea12c0-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.336663 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.336662    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.336887 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.336745    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9611d1df-c911-4812-8910-e3e530ea12c0-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.336887 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.336787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpzm\" (UniqueName: \"kubernetes.io/projected/9611d1df-c911-4812-8910-e3e530ea12c0-kube-api-access-kdpzm\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.336887 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.336819    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.438112 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.438001    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpzm\" (UniqueName: \"kubernetes.io/projected/9611d1df-c911-4812-8910-e3e530ea12c0-kube-api-access-kdpzm\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.438112 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.438073    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.438297 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.438119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9611d1df-c911-4812-8910-e3e530ea12c0-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.438297 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.438139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.438297 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.438182    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9611d1df-c911-4812-8910-e3e530ea12c0-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.438641 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.438615    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9611d1df-c911-4812-8910-e3e530ea12c0-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.438793 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.438773    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.438928 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.438884    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.440821 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.440800    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9611d1df-c911-4812-8910-e3e530ea12c0-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.446110 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.446087    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpzm\" (UniqueName: \"kubernetes.io/projected/9611d1df-c911-4812-8910-e3e530ea12c0-kube-api-access-kdpzm\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.483866 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.483832    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:20.758330 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:20.758305    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"]
Sep 08 13:59:21.395578 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:21.395540    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" event={"ID":"9611d1df-c911-4812-8910-e3e530ea12c0","Type":"ContainerStarted","Data":"d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280"}
Sep 08 13:59:21.395974 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:21.395639    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" event={"ID":"9611d1df-c911-4812-8910-e3e530ea12c0","Type":"ContainerStarted","Data":"359bb9a5341067772a8541139dd6c3599dc8fe2a7acaf7f22ff99ebe8a13de4b"}
Sep 08 13:59:23.198909 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:23.198873    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 08 13:59:25.414071 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:25.414035    2583 generic.go:356] "Generic (PLEG): container finished" podID="9611d1df-c911-4812-8910-e3e530ea12c0" containerID="d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280" exitCode=0
Sep 08 13:59:25.414071 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:25.414047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" event={"ID":"9611d1df-c911-4812-8910-e3e530ea12c0","Type":"ContainerDied","Data":"d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280"}
Sep 08 13:59:26.424912 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:26.424879    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" event={"ID":"9611d1df-c911-4812-8910-e3e530ea12c0","Type":"ContainerStarted","Data":"ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376"}
Sep 08 13:59:26.425355 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:26.424942    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" event={"ID":"9611d1df-c911-4812-8910-e3e530ea12c0","Type":"ContainerStarted","Data":"dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf"}
Sep 08 13:59:26.425355 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:26.425162    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:26.447580 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:26.447511    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" podStartSLOduration=2.552800769 podStartE2EDuration="6.447487453s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.894686684s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:59:26.445177417 +0000 UTC m=+2595.374081190" watchObservedRunningTime="2026-09-08 13:59:26.447487453 +0000 UTC m=+2595.376391209"
Sep 08 13:59:27.430290 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:27.430249    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:27.431887 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:27.431854    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 08 13:59:28.436099 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:28.436056    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 08 13:59:33.199428 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:33.199381    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 08 13:59:33.440673 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:33.440637    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:33.441801 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:33.441778    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:40.212302 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:40.212266    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"]
Sep 08 13:59:40.212828 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:40.212519    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kserve-container" containerID="cri-o://dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf" gracePeriod=30
Sep 08 13:59:40.212828 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:40.212603    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kube-rbac-proxy" containerID="cri-o://ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376" gracePeriod=30
Sep 08 13:59:40.224155 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:40.224101    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:40.495349 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:40.495254    2583 generic.go:356] "Generic (PLEG): container finished" podID="9611d1df-c911-4812-8910-e3e530ea12c0" containerID="ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376" exitCode=2
Sep 08 13:59:40.495349 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:40.495328    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" event={"ID":"9611d1df-c911-4812-8910-e3e530ea12c0","Type":"ContainerDied","Data":"ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376"}
Sep 08 13:59:40.956428 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:40.956403    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:41.021945 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.021915    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-kube-rbac-proxy-sar-config\") pod \"9611d1df-c911-4812-8910-e3e530ea12c0\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") "
Sep 08 13:59:41.022158 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.021960    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9611d1df-c911-4812-8910-e3e530ea12c0-kserve-provision-location\") pod \"9611d1df-c911-4812-8910-e3e530ea12c0\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") "
Sep 08 13:59:41.022158 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.021992    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-cabundle-cert\") pod \"9611d1df-c911-4812-8910-e3e530ea12c0\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") "
Sep 08 13:59:41.022158 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.022129    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9611d1df-c911-4812-8910-e3e530ea12c0-proxy-tls\") pod \"9611d1df-c911-4812-8910-e3e530ea12c0\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") "
Sep 08 13:59:41.022344 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.022161    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdpzm\" (UniqueName: \"kubernetes.io/projected/9611d1df-c911-4812-8910-e3e530ea12c0-kube-api-access-kdpzm\") pod \"9611d1df-c911-4812-8910-e3e530ea12c0\" (UID: \"9611d1df-c911-4812-8910-e3e530ea12c0\") "
Sep 08 13:59:41.022344 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.022268    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9611d1df-c911-4812-8910-e3e530ea12c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9611d1df-c911-4812-8910-e3e530ea12c0" (UID: "9611d1df-c911-4812-8910-e3e530ea12c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:41.022442 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.022345    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9611d1df-c911-4812-8910-e3e530ea12c0" (UID: "9611d1df-c911-4812-8910-e3e530ea12c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:59:41.022442 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.022353    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9611d1df-c911-4812-8910-e3e530ea12c0" (UID: "9611d1df-c911-4812-8910-e3e530ea12c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:59:41.022544 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.022506    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:41.022544 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.022525    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9611d1df-c911-4812-8910-e3e530ea12c0-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:41.022544 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.022540    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9611d1df-c911-4812-8910-e3e530ea12c0-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:41.024378 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.024353    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9611d1df-c911-4812-8910-e3e530ea12c0-kube-api-access-kdpzm" (OuterVolumeSpecName: "kube-api-access-kdpzm") pod "9611d1df-c911-4812-8910-e3e530ea12c0" (UID: "9611d1df-c911-4812-8910-e3e530ea12c0"). InnerVolumeSpecName "kube-api-access-kdpzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:59:41.024501 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.024451    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9611d1df-c911-4812-8910-e3e530ea12c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9611d1df-c911-4812-8910-e3e530ea12c0" (UID: "9611d1df-c911-4812-8910-e3e530ea12c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:59:41.123223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.123139    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9611d1df-c911-4812-8910-e3e530ea12c0-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:41.123223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.123168    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdpzm\" (UniqueName: \"kubernetes.io/projected/9611d1df-c911-4812-8910-e3e530ea12c0-kube-api-access-kdpzm\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:41.501914 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.501827    2583 generic.go:356] "Generic (PLEG): container finished" podID="9611d1df-c911-4812-8910-e3e530ea12c0" containerID="dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf" exitCode=0
Sep 08 13:59:41.501914 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.501888    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"
Sep 08 13:59:41.502414 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.501914    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" event={"ID":"9611d1df-c911-4812-8910-e3e530ea12c0","Type":"ContainerDied","Data":"dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf"}
Sep 08 13:59:41.502414 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.501992    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz" event={"ID":"9611d1df-c911-4812-8910-e3e530ea12c0","Type":"ContainerDied","Data":"359bb9a5341067772a8541139dd6c3599dc8fe2a7acaf7f22ff99ebe8a13de4b"}
Sep 08 13:59:41.502414 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.502055    2583 scope.go:117] "RemoveContainer" containerID="ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376"
Sep 08 13:59:41.510716 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.510702    2583 scope.go:117] "RemoveContainer" containerID="dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf"
Sep 08 13:59:41.518507 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.518488    2583 scope.go:117] "RemoveContainer" containerID="d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280"
Sep 08 13:59:41.524108 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.524083    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"]
Sep 08 13:59:41.527001 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.526982    2583 scope.go:117] "RemoveContainer" containerID="ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376"
Sep 08 13:59:41.527402 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:41.527383    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376\": container with ID starting with ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376 not found: ID does not exist" containerID="ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376"
Sep 08 13:59:41.527461 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.527411    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376"} err="failed to get container status \"ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376\": rpc error: code = NotFound desc = could not find container \"ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376\": container with ID starting with ad4da868a742805a6d32603fcf9835a788a7c68bba2dfc9f4fc08165a44d5376 not found: ID does not exist"
Sep 08 13:59:41.527461 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.527431    2583 scope.go:117] "RemoveContainer" containerID="dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf"
Sep 08 13:59:41.527578 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.527558    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-v4sfz"]
Sep 08 13:59:41.527695 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:41.527675    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf\": container with ID starting with dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf not found: ID does not exist" containerID="dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf"
Sep 08 13:59:41.527738 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.527702    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf"} err="failed to get container status \"dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf\": rpc error: code = NotFound desc = could not find container \"dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf\": container with ID starting with dd769e5ba4f3924f1ae4cddcf8f51bdbd1c1d4a71184c4c1a61af37c2a0b4ccf not found: ID does not exist"
Sep 08 13:59:41.527738 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.527716    2583 scope.go:117] "RemoveContainer" containerID="d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280"
Sep 08 13:59:41.527939 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:41.527920    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280\": container with ID starting with d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280 not found: ID does not exist" containerID="d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280"
Sep 08 13:59:41.527989 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.527946    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280"} err="failed to get container status \"d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280\": rpc error: code = NotFound desc = could not find container \"d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280\": container with ID starting with d011ffa541ccf7ad5581240141027fa95d07cc404752d6236eaac34caf482280 not found: ID does not exist"
Sep 08 13:59:41.599564 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:41.599531    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" path="/var/lib/kubelet/pods/9611d1df-c911-4812-8910-e3e530ea12c0/volumes"
Sep 08 13:59:43.199678 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:43.199643    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:59:45.391298 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.391260    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:45.420154 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.420123    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"]
Sep 08 13:59:45.420508 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.420495    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kube-rbac-proxy"
Sep 08 13:59:45.420553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.420513    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kube-rbac-proxy"
Sep 08 13:59:45.420553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.420527    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kserve-container"
Sep 08 13:59:45.420553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.420533    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kserve-container"
Sep 08 13:59:45.420553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.420540    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="storage-initializer"
Sep 08 13:59:45.420553 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.420548    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="storage-initializer"
Sep 08 13:59:45.420707 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.420620    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kserve-container"
Sep 08 13:59:45.420707 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.420630    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9611d1df-c911-4812-8910-e3e530ea12c0" containerName="kube-rbac-proxy"
Sep 08 13:59:45.425676 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.425654    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.427562 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.427540    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 08 13:59:45.427705 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.427560    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 08 13:59:45.436310 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.436282    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"]
Sep 08 13:59:45.460074 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.460035    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.460251 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.460124    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.460251 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.460148    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5j5t\" (UniqueName: \"kubernetes.io/projected/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-api-access-m5j5t\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.460251 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.460172    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.460251 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.460196    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.552729 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.552693    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:45.560751 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.560727    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5j5t\" (UniqueName: \"kubernetes.io/projected/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-api-access-m5j5t\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.560876 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.560763    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.560876 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.560787    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.560876 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.560817    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.561050 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.560910    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.561223 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.561205    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.561565 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.561543    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.561677 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.561545    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.563502 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.563479    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.572246 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.572225    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5j5t\" (UniqueName: \"kubernetes.io/projected/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-api-access-m5j5t\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:45.738089 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:45.737989    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:46.015418 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:46.015391    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"]
Sep 08 13:59:46.017504 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:59:46.017479    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4f8e8b3_57ff_4f1a_a71c_48933a62018c.slice/crio-6e12bb6ba6861a447066d91a271686cfc7fffadc8b583416bbc798bda6038509 WatchSource:0}: Error finding container 6e12bb6ba6861a447066d91a271686cfc7fffadc8b583416bbc798bda6038509: Status 404 returned error can't find the container with id 6e12bb6ba6861a447066d91a271686cfc7fffadc8b583416bbc798bda6038509
Sep 08 13:59:46.529077 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:46.529044    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" event={"ID":"d4f8e8b3-57ff-4f1a-a71c-48933a62018c","Type":"ContainerStarted","Data":"d6839ad3244b9a322d0f72de3a929328da0fc4ee844c5d22342433c167599197"}
Sep 08 13:59:46.529451 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:46.529101    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" event={"ID":"d4f8e8b3-57ff-4f1a-a71c-48933a62018c","Type":"ContainerStarted","Data":"6e12bb6ba6861a447066d91a271686cfc7fffadc8b583416bbc798bda6038509"}
Sep 08 13:59:47.800167 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:47.800136    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"]
Sep 08 13:59:47.800574 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:47.800377    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container" containerID="cri-o://0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0" gracePeriod=30
Sep 08 13:59:47.800574 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:47.800393    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kube-rbac-proxy" containerID="cri-o://924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc" gracePeriod=30
Sep 08 13:59:47.811994 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:47.811971    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:48.195000 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:48.194961    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 08 13:59:48.547043 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:48.546934    2583 generic.go:356] "Generic (PLEG): container finished" podID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerID="924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc" exitCode=2
Sep 08 13:59:48.547043 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:48.547029    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" event={"ID":"26a7309e-d080-435f-98a1-ab951b1e9a74","Type":"ContainerDied","Data":"924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc"}
Sep 08 13:59:50.558199 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:50.558111    2583 generic.go:356] "Generic (PLEG): container finished" podID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerID="d6839ad3244b9a322d0f72de3a929328da0fc4ee844c5d22342433c167599197" exitCode=0
Sep 08 13:59:50.558674 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:50.558193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" event={"ID":"d4f8e8b3-57ff-4f1a-a71c-48933a62018c","Type":"ContainerDied","Data":"d6839ad3244b9a322d0f72de3a929328da0fc4ee844c5d22342433c167599197"}
Sep 08 13:59:51.460660 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.460636    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:59:51.513894 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.513815    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a7309e-d080-435f-98a1-ab951b1e9a74-proxy-tls\") pod \"26a7309e-d080-435f-98a1-ab951b1e9a74\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") "
Sep 08 13:59:51.513894 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.513875    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-rbac-proxy-sar-config\") pod \"26a7309e-d080-435f-98a1-ab951b1e9a74\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") "
Sep 08 13:59:51.514134 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.513915    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-api-access-gn9cw\") pod \"26a7309e-d080-435f-98a1-ab951b1e9a74\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") "
Sep 08 13:59:51.514134 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.513968    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a7309e-d080-435f-98a1-ab951b1e9a74-kserve-provision-location\") pod \"26a7309e-d080-435f-98a1-ab951b1e9a74\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") "
Sep 08 13:59:51.514134 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.514052    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-cabundle-cert\") pod \"26a7309e-d080-435f-98a1-ab951b1e9a74\" (UID: \"26a7309e-d080-435f-98a1-ab951b1e9a74\") "
Sep 08 13:59:51.514403 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.514363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26a7309e-d080-435f-98a1-ab951b1e9a74" (UID: "26a7309e-d080-435f-98a1-ab951b1e9a74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:59:51.514403 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.514360    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a7309e-d080-435f-98a1-ab951b1e9a74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26a7309e-d080-435f-98a1-ab951b1e9a74" (UID: "26a7309e-d080-435f-98a1-ab951b1e9a74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 13:59:51.514813 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.514399    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26a7309e-d080-435f-98a1-ab951b1e9a74" (UID: "26a7309e-d080-435f-98a1-ab951b1e9a74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 13:59:51.516293 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.516261    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a7309e-d080-435f-98a1-ab951b1e9a74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26a7309e-d080-435f-98a1-ab951b1e9a74" (UID: "26a7309e-d080-435f-98a1-ab951b1e9a74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 13:59:51.516383 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.516297    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-api-access-gn9cw" (OuterVolumeSpecName: "kube-api-access-gn9cw") pod "26a7309e-d080-435f-98a1-ab951b1e9a74" (UID: "26a7309e-d080-435f-98a1-ab951b1e9a74"). InnerVolumeSpecName "kube-api-access-gn9cw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 13:59:51.564770 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.564733    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" event={"ID":"d4f8e8b3-57ff-4f1a-a71c-48933a62018c","Type":"ContainerStarted","Data":"6e33a760f025c680380e779557d076cc8a1b224c7370f5cb4480f27306e92283"}
Sep 08 13:59:51.565210 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.564849    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" event={"ID":"d4f8e8b3-57ff-4f1a-a71c-48933a62018c","Type":"ContainerStarted","Data":"debf68aaca2f02e36eae5c98c3d30756c2307cd4444bbc36f555c8284d0764ec"}
Sep 08 13:59:51.565210 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.565179    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:51.565336 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.565232    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:51.566864 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.566833    2583 generic.go:356] "Generic (PLEG): container finished" podID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerID="0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0" exitCode=0
Sep 08 13:59:51.566983 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.566878    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"
Sep 08 13:59:51.566983 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.566956    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" event={"ID":"26a7309e-d080-435f-98a1-ab951b1e9a74","Type":"ContainerDied","Data":"0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0"}
Sep 08 13:59:51.567089 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.567047    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p" event={"ID":"26a7309e-d080-435f-98a1-ab951b1e9a74","Type":"ContainerDied","Data":"0e8b503bb6014f1942c80af7a7536b984abc85db6894d25fb10e8dc54620804f"}
Sep 08 13:59:51.567135 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.567093    2583 scope.go:117] "RemoveContainer" containerID="924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc"
Sep 08 13:59:51.577629 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.577612    2583 scope.go:117] "RemoveContainer" containerID="0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0"
Sep 08 13:59:51.585404 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.585386    2583 scope.go:117] "RemoveContainer" containerID="d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4"
Sep 08 13:59:51.588873 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.588823    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" podStartSLOduration=2.576990039 podStartE2EDuration="6.588810087s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.011820048s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:59:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:59:51.586439773 +0000 UTC m=+2620.515343519" watchObservedRunningTime="2026-09-08 13:59:51.588810087 +0000 UTC m=+2620.517713830"
Sep 08 13:59:51.593869 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.593852    2583 scope.go:117] "RemoveContainer" containerID="924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc"
Sep 08 13:59:51.594242 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:51.594217    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc\": container with ID starting with 924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc not found: ID does not exist" containerID="924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc"
Sep 08 13:59:51.594322 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.594252    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc"} err="failed to get container status \"924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc\": rpc error: code = NotFound desc = could not find container \"924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc\": container with ID starting with 924b8e6514d381a89dfe22824b4e24b24ea01e8cde51ea82c1461cbaf35db9fc not found: ID does not exist"
Sep 08 13:59:51.594322 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.594276    2583 scope.go:117] "RemoveContainer" containerID="0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0"
Sep 08 13:59:51.594547 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:51.594529    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0\": container with ID starting with 0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0 not found: ID does not exist" containerID="0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0"
Sep 08 13:59:51.594600 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.594552    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0"} err="failed to get container status \"0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0\": rpc error: code = NotFound desc = could not find container \"0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0\": container with ID starting with 0653a83e8ae5f08573e6d4bba5b2cda0c070da310c88c1b531b98235c3138fc0 not found: ID does not exist"
Sep 08 13:59:51.594600 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.594566    2583 scope.go:117] "RemoveContainer" containerID="d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4"
Sep 08 13:59:51.594828 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:51.594813    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4\": container with ID starting with d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4 not found: ID does not exist" containerID="d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4"
Sep 08 13:59:51.594874 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.594836    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4"} err="failed to get container status \"d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4\": rpc error: code = NotFound desc = could not find container \"d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4\": container with ID starting with d36d38ece4f3d94b5808184fc620796c4ff473b8b168ca0fd6d7fd029829d2d4 not found: ID does not exist"
Sep 08 13:59:51.600874 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.600854    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"]
Sep 08 13:59:51.606321 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.606301    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-2ml6p"]
Sep 08 13:59:51.614790 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.614770    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a7309e-d080-435f-98a1-ab951b1e9a74-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:51.614907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.614794    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:51.614907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.614809    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn9cw\" (UniqueName: \"kubernetes.io/projected/26a7309e-d080-435f-98a1-ab951b1e9a74-kube-api-access-gn9cw\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:51.614907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.614823    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a7309e-d080-435f-98a1-ab951b1e9a74-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:51.614907 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:51.614837    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a7309e-d080-435f-98a1-ab951b1e9a74-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 13:59:52.965163 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.965128    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:52.980226 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.980190    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"]
Sep 08 13:59:52.980619 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.980601    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="storage-initializer"
Sep 08 13:59:52.980709 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.980621    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="storage-initializer"
Sep 08 13:59:52.980709 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.980660    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container"
Sep 08 13:59:52.980709 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.980670    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container"
Sep 08 13:59:52.980709 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.980681    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kube-rbac-proxy"
Sep 08 13:59:52.980709 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.980689    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kube-rbac-proxy"
Sep 08 13:59:52.980963 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.980777    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kube-rbac-proxy"
Sep 08 13:59:52.980963 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.980792    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" containerName="kserve-container"
Sep 08 13:59:52.985332 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.985313    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:52.987485 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.987466    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 08 13:59:52.987585 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.987477    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 08 13:59:52.999190 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:52.999169    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"]
Sep 08 13:59:53.027494 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.027460    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.027643 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.027531    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2630a78-2662-45e6-a3b0-d766aa31854d-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.027643 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.027577    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2630a78-2662-45e6-a3b0-d766aa31854d-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.027643 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.027615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zddk9\" (UniqueName: \"kubernetes.io/projected/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-api-access-zddk9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.027782 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.027638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.129055 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.129019    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.129250 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.129087    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2630a78-2662-45e6-a3b0-d766aa31854d-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.129250 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.129120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2630a78-2662-45e6-a3b0-d766aa31854d-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.129250 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.129155    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zddk9\" (UniqueName: \"kubernetes.io/projected/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-api-access-zddk9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.129250 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.129178    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.129480 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:53.129250    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 08 13:59:53.129480 ip-10-0-137-66 kubenswrapper[2583]: E0908 13:59:53.129316    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2630a78-2662-45e6-a3b0-d766aa31854d-proxy-tls podName:a2630a78-2662-45e6-a3b0-d766aa31854d nodeName:}" failed. No retries permitted until 2026-09-08 13:59:53.629295763 +0000 UTC m=+2622.558199486 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a2630a78-2662-45e6-a3b0-d766aa31854d-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" (UID: "a2630a78-2662-45e6-a3b0-d766aa31854d") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 08 13:59:53.129598 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.129549    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2630a78-2662-45e6-a3b0-d766aa31854d-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.129777 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.129759    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.129813 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.129765    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.138214 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.138185    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zddk9\" (UniqueName: \"kubernetes.io/projected/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-api-access-zddk9\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.230079 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.229976    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 13:59:53.595308 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.595280    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a7309e-d080-435f-98a1-ab951b1e9a74" path="/var/lib/kubelet/pods/26a7309e-d080-435f-98a1-ab951b1e9a74/volumes"
Sep 08 13:59:53.634250 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.634217    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2630a78-2662-45e6-a3b0-d766aa31854d-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.636856 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.636828    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2630a78-2662-45e6-a3b0-d766aa31854d-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:53.895114 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:53.895029    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:54.178857 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:54.178830    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"]
Sep 08 13:59:54.181536 ip-10-0-137-66 kubenswrapper[2583]: W0908 13:59:54.181506    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2630a78_2662_45e6_a3b0_d766aa31854d.slice/crio-4d1874ce7436917f22c517b852d5aa3188626307440ba08e67dc648cb5477082 WatchSource:0}: Error finding container 4d1874ce7436917f22c517b852d5aa3188626307440ba08e67dc648cb5477082: Status 404 returned error can't find the container with id 4d1874ce7436917f22c517b852d5aa3188626307440ba08e67dc648cb5477082
Sep 08 13:59:54.582580 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:54.582541    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" event={"ID":"a2630a78-2662-45e6-a3b0-d766aa31854d","Type":"ContainerStarted","Data":"e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397"}
Sep 08 13:59:54.582788 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:54.582605    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" event={"ID":"a2630a78-2662-45e6-a3b0-d766aa31854d","Type":"ContainerStarted","Data":"4d1874ce7436917f22c517b852d5aa3188626307440ba08e67dc648cb5477082"}
Sep 08 13:59:57.577281 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:57.577245    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 13:59:58.604078 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:58.604039    2583 generic.go:356] "Generic (PLEG): container finished" podID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerID="e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397" exitCode=0
Sep 08 13:59:58.604454 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:58.604100    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" event={"ID":"a2630a78-2662-45e6-a3b0-d766aa31854d","Type":"ContainerDied","Data":"e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397"}
Sep 08 13:59:59.610059 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:59.610030    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" event={"ID":"a2630a78-2662-45e6-a3b0-d766aa31854d","Type":"ContainerStarted","Data":"78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095"}
Sep 08 13:59:59.610444 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:59.610093    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" event={"ID":"a2630a78-2662-45e6-a3b0-d766aa31854d","Type":"ContainerStarted","Data":"f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a"}
Sep 08 13:59:59.610444 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:59.610304    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 13:59:59.631329 ip-10-0-137-66 kubenswrapper[2583]: I0908 13:59:59.630917    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" podStartSLOduration=3.677966134 podStartE2EDuration="7.630896181s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.952930047s" isStatefulPod=false podCreationTimestamp="2026-09-08 13:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 13:59:59.630788256 +0000 UTC m=+2628.559692005" watchObservedRunningTime="2026-09-08 13:59:59.630896181 +0000 UTC m=+2628.559799923"
Sep 08 14:00:00.615710 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:00.615678    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 14:00:06.626506 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:06.626476    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 14:00:23.255988 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:23.255945    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:27.581242 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:27.581213    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 14:00:35.484922 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:35.478005    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"]
Sep 08 14:00:35.484922 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:35.478504    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kserve-container" containerID="cri-o://debf68aaca2f02e36eae5c98c3d30756c2307cd4444bbc36f555c8284d0764ec" gracePeriod=30
Sep 08 14:00:35.484922 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:35.478790    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kube-rbac-proxy" containerID="cri-o://6e33a760f025c680380e779557d076cc8a1b224c7370f5cb4480f27306e92283" gracePeriod=30
Sep 08 14:00:35.495428 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:35.495395    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:35.772779 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:35.772693    2583 generic.go:356] "Generic (PLEG): container finished" podID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerID="6e33a760f025c680380e779557d076cc8a1b224c7370f5cb4480f27306e92283" exitCode=2
Sep 08 14:00:35.772779 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:35.772757    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" event={"ID":"d4f8e8b3-57ff-4f1a-a71c-48933a62018c","Type":"ContainerDied","Data":"6e33a760f025c680380e779557d076cc8a1b224c7370f5cb4480f27306e92283"}
Sep 08 14:00:36.712260 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:36.712213    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 08 14:00:37.572769 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:37.572726    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 08 14:00:38.619185 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:38.619134    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.51:8080/v2/models/isvc-mlflow-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 08 14:00:39.794299 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.794269    2583 generic.go:356] "Generic (PLEG): container finished" podID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerID="debf68aaca2f02e36eae5c98c3d30756c2307cd4444bbc36f555c8284d0764ec" exitCode=0
Sep 08 14:00:39.794641 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.794341    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" event={"ID":"d4f8e8b3-57ff-4f1a-a71c-48933a62018c","Type":"ContainerDied","Data":"debf68aaca2f02e36eae5c98c3d30756c2307cd4444bbc36f555c8284d0764ec"}
Sep 08 14:00:39.828375 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.828348    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 14:00:39.946086 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.945971    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-rbac-proxy-sar-config\") pod \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") "
Sep 08 14:00:39.946086 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.946074    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kserve-provision-location\") pod \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") "
Sep 08 14:00:39.946324 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.946099    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-proxy-tls\") pod \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") "
Sep 08 14:00:39.946324 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.946124    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-cabundle-cert\") pod \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") "
Sep 08 14:00:39.946324 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.946169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5j5t\" (UniqueName: \"kubernetes.io/projected/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-api-access-m5j5t\") pod \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\" (UID: \"d4f8e8b3-57ff-4f1a-a71c-48933a62018c\") "
Sep 08 14:00:39.946559 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.946400    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4f8e8b3-57ff-4f1a-a71c-48933a62018c" (UID: "d4f8e8b3-57ff-4f1a-a71c-48933a62018c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:39.946559 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.946475    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4f8e8b3-57ff-4f1a-a71c-48933a62018c" (UID: "d4f8e8b3-57ff-4f1a-a71c-48933a62018c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:00:39.946559 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.946482    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4f8e8b3-57ff-4f1a-a71c-48933a62018c" (UID: "d4f8e8b3-57ff-4f1a-a71c-48933a62018c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:00:39.948342 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.948318    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-api-access-m5j5t" (OuterVolumeSpecName: "kube-api-access-m5j5t") pod "d4f8e8b3-57ff-4f1a-a71c-48933a62018c" (UID: "d4f8e8b3-57ff-4f1a-a71c-48933a62018c"). InnerVolumeSpecName "kube-api-access-m5j5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:00:39.948342 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:39.948326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4f8e8b3-57ff-4f1a-a71c-48933a62018c" (UID: "d4f8e8b3-57ff-4f1a-a71c-48933a62018c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:00:40.047412 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.047379    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.047412 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.047406    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.047412 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.047416    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.047662 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.047428    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.047662 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.047437    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5j5t\" (UniqueName: \"kubernetes.io/projected/d4f8e8b3-57ff-4f1a-a71c-48933a62018c-kube-api-access-m5j5t\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:00:40.633462 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.633400    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:40.682962 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.682928    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"]
Sep 08 14:00:40.683398 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.683379    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="storage-initializer"
Sep 08 14:00:40.683502 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.683401    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="storage-initializer"
Sep 08 14:00:40.683502 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.683416    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kserve-container"
Sep 08 14:00:40.683502 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.683426    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kserve-container"
Sep 08 14:00:40.683502 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.683439    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kube-rbac-proxy"
Sep 08 14:00:40.683502 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.683447    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kube-rbac-proxy"
Sep 08 14:00:40.683795 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.683544    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kserve-container"
Sep 08 14:00:40.683795 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.683558    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" containerName="kube-rbac-proxy"
Sep 08 14:00:40.687291 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.687269    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.689317 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.689291    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 08 14:00:40.689420 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.689296    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 08 14:00:40.699945 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.699923    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"]
Sep 08 14:00:40.753209 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.753182    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aafb02c4-5963-4a7c-b8d9-497acec144b8-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.753403 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.753216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.753403 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.753267    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aafb02c4-5963-4a7c-b8d9-497acec144b8-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.753403 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.753378    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.753579 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.753426    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657kr\" (UniqueName: \"kubernetes.io/projected/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-api-access-657kr\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.806407 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.806366    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv" event={"ID":"d4f8e8b3-57ff-4f1a-a71c-48933a62018c","Type":"ContainerDied","Data":"6e12bb6ba6861a447066d91a271686cfc7fffadc8b583416bbc798bda6038509"}
Sep 08 14:00:40.806407 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.806385    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"
Sep 08 14:00:40.806988 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.806458    2583 scope.go:117] "RemoveContainer" containerID="6e33a760f025c680380e779557d076cc8a1b224c7370f5cb4480f27306e92283"
Sep 08 14:00:40.815617 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.815596    2583 scope.go:117] "RemoveContainer" containerID="debf68aaca2f02e36eae5c98c3d30756c2307cd4444bbc36f555c8284d0764ec"
Sep 08 14:00:40.823905 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.823887    2583 scope.go:117] "RemoveContainer" containerID="d6839ad3244b9a322d0f72de3a929328da0fc4ee844c5d22342433c167599197"
Sep 08 14:00:40.828750 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.828729    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"]
Sep 08 14:00:40.834639 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.834618    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-mdxrv"]
Sep 08 14:00:40.841480 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.841453    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:40.854676 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.854653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aafb02c4-5963-4a7c-b8d9-497acec144b8-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.854777 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.854693    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.854777 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.854723    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657kr\" (UniqueName: \"kubernetes.io/projected/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-api-access-657kr\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.854910 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.854778    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aafb02c4-5963-4a7c-b8d9-497acec144b8-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.854910 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.854812    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.855114 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.855095    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aafb02c4-5963-4a7c-b8d9-497acec144b8-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.855498 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.855475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.855734 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.855475    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.857360 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.857337    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aafb02c4-5963-4a7c-b8d9-497acec144b8-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.864478 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.864454    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-657kr\" (UniqueName: \"kubernetes.io/projected/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-api-access-657kr\") pod \"isvc-sklearn-mcp-predictor-69b55858dd-fr9nt\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:40.997634 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:40.997528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:41.275946 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:41.275871    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"]
Sep 08 14:00:41.278394 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:00:41.278367    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafb02c4_5963_4a7c_b8d9_497acec144b8.slice/crio-e723ece92d7870a439f133b8f635d0614679db0583c63b2d1265550f5338c417 WatchSource:0}: Error finding container e723ece92d7870a439f133b8f635d0614679db0583c63b2d1265550f5338c417: Status 404 returned error can't find the container with id e723ece92d7870a439f133b8f635d0614679db0583c63b2d1265550f5338c417
Sep 08 14:00:41.595118 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:41.595089    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4f8e8b3-57ff-4f1a-a71c-48933a62018c" path="/var/lib/kubelet/pods/d4f8e8b3-57ff-4f1a-a71c-48933a62018c/volumes"
Sep 08 14:00:41.813981 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:41.813940    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerStarted","Data":"5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5"}
Sep 08 14:00:41.814407 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:41.814026    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerStarted","Data":"e723ece92d7870a439f133b8f635d0614679db0583c63b2d1265550f5338c417"}
Sep 08 14:00:45.833349 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:45.833313    2583 generic.go:356] "Generic (PLEG): container finished" podID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerID="5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5" exitCode=0
Sep 08 14:00:45.833740 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:45.833391    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerDied","Data":"5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5"}
Sep 08 14:00:46.630170 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:46.629875    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 14:00:46.842590 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:46.842546    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerStarted","Data":"45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21"}
Sep 08 14:00:49.858557 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:49.858531    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerStarted","Data":"2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf"}
Sep 08 14:00:50.866614 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:50.866583    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerStarted","Data":"77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18"}
Sep 08 14:00:50.867069 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:50.866759    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:50.895288 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:50.895219    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podStartSLOduration=3.167989654 podStartE2EDuration="10.895205686s" totalImagesPullingTime="3.848753416s" totalInitContainerRuntime="3.878462616s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:00:50.892445276 +0000 UTC m=+2679.821349017" watchObservedRunningTime="2026-09-08 14:00:50.895205686 +0000 UTC m=+2679.824109427"
Sep 08 14:00:51.871301 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:51.871273    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:51.871663 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:51.871352    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:53.063570 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:53.063528    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"]
Sep 08 14:00:53.063975 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:53.063791    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kserve-container" containerID="cri-o://f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a" gracePeriod=30
Sep 08 14:00:53.063975 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:53.063791    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kube-rbac-proxy" containerID="cri-o://78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095" gracePeriod=30
Sep 08 14:00:53.282932 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:53.282893    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:53.882410 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:53.882382    2583 generic.go:356] "Generic (PLEG): container finished" podID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerID="78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095" exitCode=2
Sep 08 14:00:53.882584 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:53.882468    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" event={"ID":"a2630a78-2662-45e6-a3b0-d766aa31854d","Type":"ContainerDied","Data":"78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095"}
Sep 08 14:00:56.621850 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:56.621809    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 08 14:00:57.881572 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:57.881542    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:00:58.217204 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:58.217100    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:00:58.314340 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:00:58.314300    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:01:00.813737 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.813709    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 14:01:00.915316 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.915219    2583 generic.go:356] "Generic (PLEG): container finished" podID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerID="f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a" exitCode=0
Sep 08 14:01:00.915316 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.915276    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"
Sep 08 14:01:00.915316 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.915288    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" event={"ID":"a2630a78-2662-45e6-a3b0-d766aa31854d","Type":"ContainerDied","Data":"f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a"}
Sep 08 14:01:00.915580 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.915343    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt" event={"ID":"a2630a78-2662-45e6-a3b0-d766aa31854d","Type":"ContainerDied","Data":"4d1874ce7436917f22c517b852d5aa3188626307440ba08e67dc648cb5477082"}
Sep 08 14:01:00.915580 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.915371    2583 scope.go:117] "RemoveContainer" containerID="78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095"
Sep 08 14:01:00.925814 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.925794    2583 scope.go:117] "RemoveContainer" containerID="f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a"
Sep 08 14:01:00.933317 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.933299    2583 scope.go:117] "RemoveContainer" containerID="e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397"
Sep 08 14:01:00.933438 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.933421    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2630a78-2662-45e6-a3b0-d766aa31854d-proxy-tls\") pod \"a2630a78-2662-45e6-a3b0-d766aa31854d\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") "
Sep 08 14:01:00.933483 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.933459    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zddk9\" (UniqueName: \"kubernetes.io/projected/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-api-access-zddk9\") pod \"a2630a78-2662-45e6-a3b0-d766aa31854d\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") "
Sep 08 14:01:00.933531 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.933491    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-rbac-proxy-sar-config\") pod \"a2630a78-2662-45e6-a3b0-d766aa31854d\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") "
Sep 08 14:01:00.933622 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.933607    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-cabundle-cert\") pod \"a2630a78-2662-45e6-a3b0-d766aa31854d\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") "
Sep 08 14:01:00.933682 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.933641    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2630a78-2662-45e6-a3b0-d766aa31854d-kserve-provision-location\") pod \"a2630a78-2662-45e6-a3b0-d766aa31854d\" (UID: \"a2630a78-2662-45e6-a3b0-d766aa31854d\") "
Sep 08 14:01:00.934127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.933966    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a2630a78-2662-45e6-a3b0-d766aa31854d" (UID: "a2630a78-2662-45e6-a3b0-d766aa31854d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:01:00.934127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.934102    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2630a78-2662-45e6-a3b0-d766aa31854d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a2630a78-2662-45e6-a3b0-d766aa31854d" (UID: "a2630a78-2662-45e6-a3b0-d766aa31854d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:01:00.934271 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.934129    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a2630a78-2662-45e6-a3b0-d766aa31854d" (UID: "a2630a78-2662-45e6-a3b0-d766aa31854d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:01:00.935687 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.935662    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2630a78-2662-45e6-a3b0-d766aa31854d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a2630a78-2662-45e6-a3b0-d766aa31854d" (UID: "a2630a78-2662-45e6-a3b0-d766aa31854d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:01:00.935687 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.935671    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-api-access-zddk9" (OuterVolumeSpecName: "kube-api-access-zddk9") pod "a2630a78-2662-45e6-a3b0-d766aa31854d" (UID: "a2630a78-2662-45e6-a3b0-d766aa31854d"). InnerVolumeSpecName "kube-api-access-zddk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:01:00.945426 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.945407    2583 scope.go:117] "RemoveContainer" containerID="78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095"
Sep 08 14:01:00.945699 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:01:00.945672    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095\": container with ID starting with 78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095 not found: ID does not exist" containerID="78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095"
Sep 08 14:01:00.945759 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.945699    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095"} err="failed to get container status \"78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095\": rpc error: code = NotFound desc = could not find container \"78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095\": container with ID starting with 78e87672f9be161a9025174f502c683037bba661639453fa2be27ee195d3a095 not found: ID does not exist"
Sep 08 14:01:00.945759 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.945721    2583 scope.go:117] "RemoveContainer" containerID="f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a"
Sep 08 14:01:00.945950 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:01:00.945937    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a\": container with ID starting with f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a not found: ID does not exist" containerID="f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a"
Sep 08 14:01:00.945990 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.945954    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a"} err="failed to get container status \"f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a\": rpc error: code = NotFound desc = could not find container \"f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a\": container with ID starting with f9ee034b5b7cf14708d71d34a9a4ee7b4871e1e54b93c7d2f32e6dc471b4ca7a not found: ID does not exist"
Sep 08 14:01:00.945990 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.945969    2583 scope.go:117] "RemoveContainer" containerID="e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397"
Sep 08 14:01:00.946189 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:01:00.946173    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397\": container with ID starting with e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397 not found: ID does not exist" containerID="e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397"
Sep 08 14:01:00.946240 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:00.946193    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397"} err="failed to get container status \"e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397\": rpc error: code = NotFound desc = could not find container \"e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397\": container with ID starting with e776cc1b9904fc2c61793016150da5f88c64e9894bbe804b0c7721d046bb9397 not found: ID does not exist"
Sep 08 14:01:01.034848 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:01.034815    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zddk9\" (UniqueName: \"kubernetes.io/projected/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-api-access-zddk9\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:01:01.034848 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:01.034845    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:01:01.034848 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:01.034854    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a2630a78-2662-45e6-a3b0-d766aa31854d-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:01:01.035118 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:01.034863    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a2630a78-2662-45e6-a3b0-d766aa31854d-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:01:01.035118 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:01.034876    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a2630a78-2662-45e6-a3b0-d766aa31854d-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:01:01.238054 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:01.238028    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"]
Sep 08 14:01:01.242464 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:01.242439    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-m5sgt"]
Sep 08 14:01:01.595588 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:01.595555    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" path="/var/lib/kubelet/pods/a2630a78-2662-45e6-a3b0-d766aa31854d/volumes"
Sep 08 14:01:11.750079 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:11.749983    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 14:01:11.753983 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:11.753965    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 14:01:17.882526 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:17.882488    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 08 14:01:20.998251 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:20.998201    2583 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 08 14:01:27.885867 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:27.885834    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:01:57.886926 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:01:57.886898    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:02:00.753974 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:00.753942    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"]
Sep 08 14:02:00.754384 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:00.754223    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-container" containerID="cri-o://45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21" gracePeriod=30
Sep 08 14:02:00.754384 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:00.754228    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy" containerID="cri-o://77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18" gracePeriod=30
Sep 08 14:02:00.754384 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:00.754226    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-agent" containerID="cri-o://2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf" gracePeriod=30
Sep 08 14:02:00.768680 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:00.768657    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:01.184343 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:01.184304    2583 generic.go:356] "Generic (PLEG): container finished" podID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerID="77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18" exitCode=2
Sep 08 14:02:01.184527 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:01.184376    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerDied","Data":"77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18"}
Sep 08 14:02:02.877249 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:02.877202    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 08 14:02:03.195122 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:03.195088    2583 generic.go:356] "Generic (PLEG): container finished" podID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerID="45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21" exitCode=0
Sep 08 14:02:03.195299 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:03.195164    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerDied","Data":"45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21"}
Sep 08 14:02:05.930600 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.930568    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"]
Sep 08 14:02:05.930977 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.930912    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kserve-container"
Sep 08 14:02:05.930977 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.930923    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kserve-container"
Sep 08 14:02:05.930977 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.930932    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kube-rbac-proxy"
Sep 08 14:02:05.930977 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.930938    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kube-rbac-proxy"
Sep 08 14:02:05.930977 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.930946    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="storage-initializer"
Sep 08 14:02:05.930977 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.930952    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="storage-initializer"
Sep 08 14:02:05.931197 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.931027    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kube-rbac-proxy"
Sep 08 14:02:05.931197 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.931039    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2630a78-2662-45e6-a3b0-d766aa31854d" containerName="kserve-container"
Sep 08 14:02:05.934445 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.934423    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:05.936441 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.936423    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 08 14:02:05.936543 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.936439    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 08 14:02:05.946578 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.946553    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"]
Sep 08 14:02:05.987195 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.987160    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8da431ce-21fc-43da-bfcf-2c97ebfe7914-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:05.987369 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.987209    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg29r\" (UniqueName: \"kubernetes.io/projected/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-api-access-rg29r\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:05.987369 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.987276    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:05.987369 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.987334    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:05.987369 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:05.987358    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.088639 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.088596    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8da431ce-21fc-43da-bfcf-2c97ebfe7914-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.088830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.088655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg29r\" (UniqueName: \"kubernetes.io/projected/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-api-access-rg29r\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.088830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.088682    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.088830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.088708    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.088830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.088722    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.089160 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.089140    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.089447 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.089425    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.089573 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.089482    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.091367 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.091348    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8da431ce-21fc-43da-bfcf-2c97ebfe7914-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.098323 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.098304    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg29r\" (UniqueName: \"kubernetes.io/projected/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-api-access-rg29r\") pod \"isvc-paddle-predictor-6d5c9fd4f6-ptc4g\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.246242 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.246150    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:06.525432 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.525358    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"]
Sep 08 14:02:06.529137 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:02:06.529108    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da431ce_21fc_43da_bfcf_2c97ebfe7914.slice/crio-5325d6c21b6068e5a11a21f5010e9f8c8b7c50af3280f07e1489ac1a11d700f4 WatchSource:0}: Error finding container 5325d6c21b6068e5a11a21f5010e9f8c8b7c50af3280f07e1489ac1a11d700f4: Status 404 returned error can't find the container with id 5325d6c21b6068e5a11a21f5010e9f8c8b7c50af3280f07e1489ac1a11d700f4
Sep 08 14:02:06.589810 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:06.589779    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:07.214386 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:07.214348    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" event={"ID":"8da431ce-21fc-43da-bfcf-2c97ebfe7914","Type":"ContainerStarted","Data":"41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095"}
Sep 08 14:02:07.214789 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:07.214434    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" event={"ID":"8da431ce-21fc-43da-bfcf-2c97ebfe7914","Type":"ContainerStarted","Data":"5325d6c21b6068e5a11a21f5010e9f8c8b7c50af3280f07e1489ac1a11d700f4"}
Sep 08 14:02:07.877515 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:07.877470    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 08 14:02:07.882944 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:07.882910    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 08 14:02:11.232923 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:11.232890    2583 generic.go:356] "Generic (PLEG): container finished" podID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerID="41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095" exitCode=0
Sep 08 14:02:11.233338 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:11.232937    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" event={"ID":"8da431ce-21fc-43da-bfcf-2c97ebfe7914","Type":"ContainerDied","Data":"41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095"}
Sep 08 14:02:12.877007 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:12.876956    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 08 14:02:12.877478 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:12.877138    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:02:17.877177 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:17.877131    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 08 14:02:17.882823 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:17.882783    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 08 14:02:18.414866 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:18.414820    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:22.877344 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:22.877316    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 08 14:02:23.299217 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:23.299131    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" event={"ID":"8da431ce-21fc-43da-bfcf-2c97ebfe7914","Type":"ContainerStarted","Data":"9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb"}
Sep 08 14:02:23.299217 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:23.299195    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" event={"ID":"8da431ce-21fc-43da-bfcf-2c97ebfe7914","Type":"ContainerStarted","Data":"729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d"}
Sep 08 14:02:23.299478 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:23.299460    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:23.299610 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:23.299595    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:23.300983 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:23.300957    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 08 14:02:23.318472 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:23.318426    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podStartSLOduration=2.277109964 podStartE2EDuration="18.318411679s" totalImagesPullingTime="11.63849142s" totalInitContainerRuntime="4.402810295s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:02:23.317267541 +0000 UTC m=+2772.246171283" watchObservedRunningTime="2026-09-08 14:02:23.318411679 +0000 UTC m=+2772.247315420"
Sep 08 14:02:23.478837 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:23.478796    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:23.731975 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:23.731938    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:02:24.303857 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:24.303816    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 08 14:02:27.876762 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:27.876716    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 08 14:02:27.882788 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:27.882759    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.53:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 08 14:02:27.882892 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:27.882880    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:02:29.308215 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:29.308188    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:02:29.308825 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:29.308646    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 08 14:02:30.911845 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:30.911812    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:02:31.026404 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.026318    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aafb02c4-5963-4a7c-b8d9-497acec144b8-proxy-tls\") pod \"aafb02c4-5963-4a7c-b8d9-497acec144b8\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") "
Sep 08 14:02:31.026404 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.026360    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aafb02c4-5963-4a7c-b8d9-497acec144b8-kserve-provision-location\") pod \"aafb02c4-5963-4a7c-b8d9-497acec144b8\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") "
Sep 08 14:02:31.026404 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.026379    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-cabundle-cert\") pod \"aafb02c4-5963-4a7c-b8d9-497acec144b8\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") "
Sep 08 14:02:31.026724 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.026415    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-657kr\" (UniqueName: \"kubernetes.io/projected/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-api-access-657kr\") pod \"aafb02c4-5963-4a7c-b8d9-497acec144b8\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") "
Sep 08 14:02:31.026724 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.026523    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-rbac-proxy-sar-config\") pod \"aafb02c4-5963-4a7c-b8d9-497acec144b8\" (UID: \"aafb02c4-5963-4a7c-b8d9-497acec144b8\") "
Sep 08 14:02:31.026883 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.026711    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb02c4-5963-4a7c-b8d9-497acec144b8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aafb02c4-5963-4a7c-b8d9-497acec144b8" (UID: "aafb02c4-5963-4a7c-b8d9-497acec144b8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:02:31.026883 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.026802    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aafb02c4-5963-4a7c-b8d9-497acec144b8" (UID: "aafb02c4-5963-4a7c-b8d9-497acec144b8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:02:31.027024 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.026985    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aafb02c4-5963-4a7c-b8d9-497acec144b8" (UID: "aafb02c4-5963-4a7c-b8d9-497acec144b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:02:31.029277 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.029240    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-api-access-657kr" (OuterVolumeSpecName: "kube-api-access-657kr") pod "aafb02c4-5963-4a7c-b8d9-497acec144b8" (UID: "aafb02c4-5963-4a7c-b8d9-497acec144b8"). InnerVolumeSpecName "kube-api-access-657kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:02:31.029399 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.029277    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aafb02c4-5963-4a7c-b8d9-497acec144b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aafb02c4-5963-4a7c-b8d9-497acec144b8" (UID: "aafb02c4-5963-4a7c-b8d9-497acec144b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:02:31.127362 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.127322    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:02:31.127362 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.127355    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aafb02c4-5963-4a7c-b8d9-497acec144b8-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:02:31.127362 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.127368    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aafb02c4-5963-4a7c-b8d9-497acec144b8-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:02:31.127602 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.127378    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aafb02c4-5963-4a7c-b8d9-497acec144b8-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:02:31.127602 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.127387    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657kr\" (UniqueName: \"kubernetes.io/projected/aafb02c4-5963-4a7c-b8d9-497acec144b8-kube-api-access-657kr\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:02:31.340648 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.340613    2583 generic.go:356] "Generic (PLEG): container finished" podID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerID="2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf" exitCode=0
Sep 08 14:02:31.340830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.340671    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerDied","Data":"2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf"}
Sep 08 14:02:31.340830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.340721    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt" event={"ID":"aafb02c4-5963-4a7c-b8d9-497acec144b8","Type":"ContainerDied","Data":"e723ece92d7870a439f133b8f635d0614679db0583c63b2d1265550f5338c417"}
Sep 08 14:02:31.340830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.340750    2583 scope.go:117] "RemoveContainer" containerID="77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18"
Sep 08 14:02:31.340830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.340673    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"
Sep 08 14:02:31.355223 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.355202    2583 scope.go:117] "RemoveContainer" containerID="2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf"
Sep 08 14:02:31.362820 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.362800    2583 scope.go:117] "RemoveContainer" containerID="45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21"
Sep 08 14:02:31.368140 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.368121    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"]
Sep 08 14:02:31.370820 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.370803    2583 scope.go:117] "RemoveContainer" containerID="5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5"
Sep 08 14:02:31.374736 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.374714    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-69b55858dd-fr9nt"]
Sep 08 14:02:31.378978 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.378961    2583 scope.go:117] "RemoveContainer" containerID="77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18"
Sep 08 14:02:31.379292 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:02:31.379262    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18\": container with ID starting with 77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18 not found: ID does not exist" containerID="77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18"
Sep 08 14:02:31.379369 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.379306    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18"} err="failed to get container status \"77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18\": rpc error: code = NotFound desc = could not find container \"77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18\": container with ID starting with 77a14a40ca6cf73201e393c7fe7807fe5310a1c32bb833ccea8182137b645e18 not found: ID does not exist"
Sep 08 14:02:31.379369 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.379333    2583 scope.go:117] "RemoveContainer" containerID="2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf"
Sep 08 14:02:31.379621 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:02:31.379604    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf\": container with ID starting with 2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf not found: ID does not exist" containerID="2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf"
Sep 08 14:02:31.379674 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.379626    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf"} err="failed to get container status \"2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf\": rpc error: code = NotFound desc = could not find container \"2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf\": container with ID starting with 2bfa8f6ab60c3eb58eee6db208a173dda62ccbe7b1e5033b0b575732ce754daf not found: ID does not exist"
Sep 08 14:02:31.379674 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.379641    2583 scope.go:117] "RemoveContainer" containerID="45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21"
Sep 08 14:02:31.379862 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:02:31.379847    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21\": container with ID starting with 45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21 not found: ID does not exist" containerID="45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21"
Sep 08 14:02:31.379902 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.379866    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21"} err="failed to get container status \"45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21\": rpc error: code = NotFound desc = could not find container \"45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21\": container with ID starting with 45ad532965d3c0a889684894f7e6f823c82a63c6b851e965598dbe3a2fd0fa21 not found: ID does not exist"
Sep 08 14:02:31.379902 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.379879    2583 scope.go:117] "RemoveContainer" containerID="5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5"
Sep 08 14:02:31.380187 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:02:31.380168    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5\": container with ID starting with 5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5 not found: ID does not exist" containerID="5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5"
Sep 08 14:02:31.380245 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.380190    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5"} err="failed to get container status \"5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5\": rpc error: code = NotFound desc = could not find container \"5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5\": container with ID starting with 5d40e12e2709d90d70744b910125fb4482f527d8d9158ac53bf76b4831b6e5e5 not found: ID does not exist"
Sep 08 14:02:31.594967 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:31.594889    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" path="/var/lib/kubelet/pods/aafb02c4-5963-4a7c-b8d9-497acec144b8/volumes"
Sep 08 14:02:39.308884 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:39.308840    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 08 14:02:49.309365 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:49.309316    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 08 14:02:59.309230 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:02:59.309189    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 08 14:03:09.309228 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:09.309196    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:03:17.060332 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:17.060296    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"]
Sep 08 14:03:17.060751 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:17.060546    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container" containerID="cri-o://729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d" gracePeriod=30
Sep 08 14:03:17.060751 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:17.060589    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kube-rbac-proxy" containerID="cri-o://9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb" gracePeriod=30
Sep 08 14:03:17.097457 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:17.097424    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:17.552300 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:17.552268    2583 generic.go:356] "Generic (PLEG): container finished" podID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerID="9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb" exitCode=2
Sep 08 14:03:17.552505 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:17.552345    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" event={"ID":"8da431ce-21fc-43da-bfcf-2c97ebfe7914","Type":"ContainerDied","Data":"9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb"}
Sep 08 14:03:19.304910 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.304868    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 08 14:03:19.309244 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.309217    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 08 14:03:19.771907 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.771884    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:03:19.862530 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.862447    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-rbac-proxy-sar-config\") pod \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") "
Sep 08 14:03:19.862530 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.862508    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8da431ce-21fc-43da-bfcf-2c97ebfe7914-proxy-tls\") pod \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") "
Sep 08 14:03:19.862760 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.862550    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-cabundle-cert\") pod \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") "
Sep 08 14:03:19.862760 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.862600    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg29r\" (UniqueName: \"kubernetes.io/projected/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-api-access-rg29r\") pod \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") "
Sep 08 14:03:19.862760 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.862712    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kserve-provision-location\") pod \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\" (UID: \"8da431ce-21fc-43da-bfcf-2c97ebfe7914\") "
Sep 08 14:03:19.862926 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.862863    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8da431ce-21fc-43da-bfcf-2c97ebfe7914" (UID: "8da431ce-21fc-43da-bfcf-2c97ebfe7914"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:03:19.863053 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.863002    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8da431ce-21fc-43da-bfcf-2c97ebfe7914" (UID: "8da431ce-21fc-43da-bfcf-2c97ebfe7914"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:03:19.863166 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.863132    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:03:19.863166 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.863150    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8da431ce-21fc-43da-bfcf-2c97ebfe7914-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:03:19.864813 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.864792    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-api-access-rg29r" (OuterVolumeSpecName: "kube-api-access-rg29r") pod "8da431ce-21fc-43da-bfcf-2c97ebfe7914" (UID: "8da431ce-21fc-43da-bfcf-2c97ebfe7914"). InnerVolumeSpecName "kube-api-access-rg29r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:03:19.865133 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.865113    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da431ce-21fc-43da-bfcf-2c97ebfe7914-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8da431ce-21fc-43da-bfcf-2c97ebfe7914" (UID: "8da431ce-21fc-43da-bfcf-2c97ebfe7914"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:03:19.870957 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.870936    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8da431ce-21fc-43da-bfcf-2c97ebfe7914" (UID: "8da431ce-21fc-43da-bfcf-2c97ebfe7914"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:03:19.963723 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.963691    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg29r\" (UniqueName: \"kubernetes.io/projected/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kube-api-access-rg29r\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:03:19.963723 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.963723    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8da431ce-21fc-43da-bfcf-2c97ebfe7914-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:03:19.963869 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:19.963735    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8da431ce-21fc-43da-bfcf-2c97ebfe7914-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:03:20.567513 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.567478    2583 generic.go:356] "Generic (PLEG): container finished" podID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerID="729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d" exitCode=0
Sep 08 14:03:20.567883 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.567536    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"
Sep 08 14:03:20.567883 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.567559    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" event={"ID":"8da431ce-21fc-43da-bfcf-2c97ebfe7914","Type":"ContainerDied","Data":"729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d"}
Sep 08 14:03:20.567883 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.567617    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g" event={"ID":"8da431ce-21fc-43da-bfcf-2c97ebfe7914","Type":"ContainerDied","Data":"5325d6c21b6068e5a11a21f5010e9f8c8b7c50af3280f07e1489ac1a11d700f4"}
Sep 08 14:03:20.567883 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.567645    2583 scope.go:117] "RemoveContainer" containerID="9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb"
Sep 08 14:03:20.576858 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.576748    2583 scope.go:117] "RemoveContainer" containerID="729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d"
Sep 08 14:03:20.584207 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.584189    2583 scope.go:117] "RemoveContainer" containerID="41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095"
Sep 08 14:03:20.590117 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.590096    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"]
Sep 08 14:03:20.592789 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.592769    2583 scope.go:117] "RemoveContainer" containerID="9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb"
Sep 08 14:03:20.593196 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:03:20.593172    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb\": container with ID starting with 9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb not found: ID does not exist" containerID="9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb"
Sep 08 14:03:20.593294 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.593204    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb"} err="failed to get container status \"9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb\": rpc error: code = NotFound desc = could not find container \"9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb\": container with ID starting with 9472ccbdfdf0aa84f9a75a53d60b087728fc437caa4f5b9209e364471978dcdb not found: ID does not exist"
Sep 08 14:03:20.593294 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.593223    2583 scope.go:117] "RemoveContainer" containerID="729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d"
Sep 08 14:03:20.593511 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:03:20.593480    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d\": container with ID starting with 729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d not found: ID does not exist" containerID="729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d"
Sep 08 14:03:20.593593 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.593507    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d"} err="failed to get container status \"729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d\": rpc error: code = NotFound desc = could not find container \"729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d\": container with ID starting with 729704c9aad2f47363185ce9152b87d1485a02fcfee6b0b5431cc3172067a76d not found: ID does not exist"
Sep 08 14:03:20.593593 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.593530    2583 scope.go:117] "RemoveContainer" containerID="41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095"
Sep 08 14:03:20.593812 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:03:20.593795    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095\": container with ID starting with 41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095 not found: ID does not exist" containerID="41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095"
Sep 08 14:03:20.593892 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.593817    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095"} err="failed to get container status \"41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095\": rpc error: code = NotFound desc = could not find container \"41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095\": container with ID starting with 41c42f18980e1e5ec7a2bd4227f77231ec217fdd19903ff4effd8198ad524095 not found: ID does not exist"
Sep 08 14:03:20.594448 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:20.594429    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-ptc4g"]
Sep 08 14:03:21.594372 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:21.594332    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" path="/var/lib/kubelet/pods/8da431ce-21fc-43da-bfcf-2c97ebfe7914/volumes"
Sep 08 14:03:22.205520 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.205484    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:22.236222 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236187    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"]
Sep 08 14:03:22.236606 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236589    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-container"
Sep 08 14:03:22.236696 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236610    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-container"
Sep 08 14:03:22.236696 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236622    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container"
Sep 08 14:03:22.236696 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236631    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container"
Sep 08 14:03:22.236696 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236651    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="storage-initializer"
Sep 08 14:03:22.236696 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236660    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="storage-initializer"
Sep 08 14:03:22.236696 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236672    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kube-rbac-proxy"
Sep 08 14:03:22.236696 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236681    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kube-rbac-proxy"
Sep 08 14:03:22.236696 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236693    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236701    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236722    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="storage-initializer"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236731    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="storage-initializer"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236748    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-agent"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236756    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-agent"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236833    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-agent"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236847    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kube-rbac-proxy"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236859    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kserve-container"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236869    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da431ce-21fc-43da-bfcf-2c97ebfe7914" containerName="kserve-container"
Sep 08 14:03:22.237127 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.236878    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="aafb02c4-5963-4a7c-b8d9-497acec144b8" containerName="kube-rbac-proxy"
Sep 08 14:03:22.240384 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.240366    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.242611 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.242589    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Sep 08 14:03:22.242611 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.242604    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 14:03:22.242861 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.242840    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Sep 08 14:03:22.242976 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.242907    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 14:03:22.243133 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.243117    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:03:22.243299 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.243286    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 14:03:22.243352 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.243322    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 14:03:22.251885 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.251865    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"]
Sep 08 14:03:22.359355 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.359318    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:22.385110 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.385081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zml\" (UniqueName: \"kubernetes.io/projected/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-api-access-99zml\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.385110 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.385110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.385264 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.385140    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.385264 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.385212    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.385264 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.385253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.486082 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.485995    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.486082 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.486044    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.486260 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.486104    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99zml\" (UniqueName: \"kubernetes.io/projected/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-api-access-99zml\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.486260 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.486120    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.486260 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.486149    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.486526 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.486503    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.486724 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.486706    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.486762 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.486735    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.488681 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.488663    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.495230 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.495212    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zml\" (UniqueName: \"kubernetes.io/projected/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-api-access-99zml\") pod \"isvc-paddle-runtime-predictor-58b99fff76-qkqbd\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.552687 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.552661    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:22.826551 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:22.826527    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"]
Sep 08 14:03:22.828921 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:03:22.828896    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8858622a_dba1_4f5d_b2eb_7f1e0c85513b.slice/crio-55d96758b0163b55eb3df0b9ac14bb3c4f96cc39795cb59e8c91dfaaeef0f059 WatchSource:0}: Error finding container 55d96758b0163b55eb3df0b9ac14bb3c4f96cc39795cb59e8c91dfaaeef0f059: Status 404 returned error can't find the container with id 55d96758b0163b55eb3df0b9ac14bb3c4f96cc39795cb59e8c91dfaaeef0f059
Sep 08 14:03:23.594134 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:23.594102    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" event={"ID":"8858622a-dba1-4f5d-b2eb-7f1e0c85513b","Type":"ContainerStarted","Data":"ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f"}
Sep 08 14:03:23.594302 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:23.594155    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" event={"ID":"8858622a-dba1-4f5d-b2eb-7f1e0c85513b","Type":"ContainerStarted","Data":"55d96758b0163b55eb3df0b9ac14bb3c4f96cc39795cb59e8c91dfaaeef0f059"}
Sep 08 14:03:27.607532 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:27.607500    2583 generic.go:356] "Generic (PLEG): container finished" podID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerID="ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f" exitCode=0
Sep 08 14:03:27.608039 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:27.607580    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" event={"ID":"8858622a-dba1-4f5d-b2eb-7f1e0c85513b","Type":"ContainerDied","Data":"ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f"}
Sep 08 14:03:28.620533 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:28.620493    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" event={"ID":"8858622a-dba1-4f5d-b2eb-7f1e0c85513b","Type":"ContainerStarted","Data":"f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490"}
Sep 08 14:03:28.620953 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:28.620582    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" event={"ID":"8858622a-dba1-4f5d-b2eb-7f1e0c85513b","Type":"ContainerStarted","Data":"946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282"}
Sep 08 14:03:28.620953 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:28.620852    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:28.621088 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:28.621034    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:28.622258 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:28.622233    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 08 14:03:28.642105 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:28.642051    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podStartSLOduration=2.259912675 podStartE2EDuration="6.642036693s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.382124018s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:03:28.640819935 +0000 UTC m=+2837.569723696" watchObservedRunningTime="2026-09-08 14:03:28.642036693 +0000 UTC m=+2837.570940432"
Sep 08 14:03:29.625864 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:29.625813    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 08 14:03:34.630357 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:34.630326    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:03:34.630877 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:34.630849    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 08 14:03:43.617891 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.617852    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:43.782989 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.782950    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:43.799205 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.799169    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"]
Sep 08 14:03:43.803156 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.803136    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.805723 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.805694    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 08 14:03:43.805831 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.805745    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 08 14:03:43.805831 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.805763    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 08 14:03:43.815875 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.815850    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"]
Sep 08 14:03:43.863053 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.862994    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42f9d47-41bb-4f06-85ae-ae0409e65e23-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.863217 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.863056    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.863217 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.863085    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.863217 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.863132    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9btbg\" (UniqueName: \"kubernetes.io/projected/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-api-access-9btbg\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.863351 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.863205    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.964453 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.964374    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42f9d47-41bb-4f06-85ae-ae0409e65e23-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.964453 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.964408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.964453 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.964430    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.964746 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:03:43.964514    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 08 14:03:43.964746 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.964554    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9btbg\" (UniqueName: \"kubernetes.io/projected/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-api-access-9btbg\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.964746 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:03:43.964604    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a42f9d47-41bb-4f06-85ae-ae0409e65e23-proxy-tls podName:a42f9d47-41bb-4f06-85ae-ae0409e65e23 nodeName:}" failed. No retries permitted until 2026-09-08 14:03:44.464568863 +0000 UTC m=+2853.393472583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a42f9d47-41bb-4f06-85ae-ae0409e65e23-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" (UID: "a42f9d47-41bb-4f06-85ae-ae0409e65e23") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 08 14:03:43.964746 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.964672    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.965123 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.965098    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.965226 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.965204    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.965339 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.965321    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:43.973759 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:43.973741    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9btbg\" (UniqueName: \"kubernetes.io/projected/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-api-access-9btbg\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:44.468887 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:44.468851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42f9d47-41bb-4f06-85ae-ae0409e65e23-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:44.471323 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:44.471306    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42f9d47-41bb-4f06-85ae-ae0409e65e23-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:44.630824 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:44.630784    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 08 14:03:44.713675 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:44.713643    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:44.743400 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:44.743359    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:03:44.989995 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:44.989970    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"]
Sep 08 14:03:44.992253 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:03:44.992228    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda42f9d47_41bb_4f06_85ae_ae0409e65e23.slice/crio-e6096a532640807b545ef9ba44de6ac88f57c4571c7c396ec01d42bf63bfebb6 WatchSource:0}: Error finding container e6096a532640807b545ef9ba44de6ac88f57c4571c7c396ec01d42bf63bfebb6: Status 404 returned error can't find the container with id e6096a532640807b545ef9ba44de6ac88f57c4571c7c396ec01d42bf63bfebb6
Sep 08 14:03:45.707414 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:45.707375    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" event={"ID":"a42f9d47-41bb-4f06-85ae-ae0409e65e23","Type":"ContainerStarted","Data":"908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f"}
Sep 08 14:03:45.707918 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:45.707435    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" event={"ID":"a42f9d47-41bb-4f06-85ae-ae0409e65e23","Type":"ContainerStarted","Data":"e6096a532640807b545ef9ba44de6ac88f57c4571c7c396ec01d42bf63bfebb6"}
Sep 08 14:03:46.713497 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:46.713458    2583 generic.go:356] "Generic (PLEG): container finished" podID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerID="908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f" exitCode=0
Sep 08 14:03:46.713867 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:46.713520    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" event={"ID":"a42f9d47-41bb-4f06-85ae-ae0409e65e23","Type":"ContainerDied","Data":"908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f"}
Sep 08 14:03:47.721597 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:47.721560    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" event={"ID":"a42f9d47-41bb-4f06-85ae-ae0409e65e23","Type":"ContainerStarted","Data":"24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf"}
Sep 08 14:03:47.722029 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:47.721625    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" event={"ID":"a42f9d47-41bb-4f06-85ae-ae0409e65e23","Type":"ContainerStarted","Data":"b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544"}
Sep 08 14:03:47.722029 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:47.721870    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:47.722029 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:47.721932    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:47.723395 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:47.723370    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 08 14:03:47.745178 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:47.745121    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podStartSLOduration=3.674220522 podStartE2EDuration="4.745105032s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.07088451s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:03:47.743490287 +0000 UTC m=+2856.672394054" watchObservedRunningTime="2026-09-08 14:03:47.745105032 +0000 UTC m=+2856.674008831"
Sep 08 14:03:48.727877 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:48.727838    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 08 14:03:53.732142 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:53.732114    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:03:53.732915 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:53.732882    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 08 14:03:54.630949 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:03:54.630908    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 08 14:04:03.733267 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:03.733183    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 08 14:04:04.631048 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:04.630987    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 08 14:04:13.732982 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:13.732939    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 08 14:04:14.632206 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:14.632169    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:04:21.976231 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:21.976195    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:23.352886 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:23.352848    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"]
Sep 08 14:04:23.353408 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:23.353171    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container" containerID="cri-o://946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282" gracePeriod=30
Sep 08 14:04:23.353408 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:23.353215    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kube-rbac-proxy" containerID="cri-o://f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490" gracePeriod=30
Sep 08 14:04:23.377473 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:23.377438    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:23.733105 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:23.732996    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 08 14:04:23.890080 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:23.890049    2583 generic.go:356] "Generic (PLEG): container finished" podID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerID="f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490" exitCode=2
Sep 08 14:04:23.890244 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:23.890122    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" event={"ID":"8858622a-dba1-4f5d-b2eb-7f1e0c85513b","Type":"ContainerDied","Data":"f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490"}
Sep 08 14:04:24.626716 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:24.626671    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Sep 08 14:04:24.631312 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:24.631289    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 08 14:04:26.121616 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.121595    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:04:26.138411 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.138389    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kserve-provision-location\") pod \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") "
Sep 08 14:04:26.138523 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.138422    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-cabundle-cert\") pod \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") "
Sep 08 14:04:26.138523 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.138475    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99zml\" (UniqueName: \"kubernetes.io/projected/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-api-access-99zml\") pod \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") "
Sep 08 14:04:26.138523 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.138496    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-proxy-tls\") pod \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") "
Sep 08 14:04:26.138659 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.138637    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-rbac-proxy-sar-config\") pod \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\" (UID: \"8858622a-dba1-4f5d-b2eb-7f1e0c85513b\") "
Sep 08 14:04:26.138913 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.138829    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8858622a-dba1-4f5d-b2eb-7f1e0c85513b" (UID: "8858622a-dba1-4f5d-b2eb-7f1e0c85513b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:04:26.139052 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.138970    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:04:26.139052 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.138987    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8858622a-dba1-4f5d-b2eb-7f1e0c85513b" (UID: "8858622a-dba1-4f5d-b2eb-7f1e0c85513b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:04:26.140678 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.140653    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8858622a-dba1-4f5d-b2eb-7f1e0c85513b" (UID: "8858622a-dba1-4f5d-b2eb-7f1e0c85513b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:04:26.140781 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.140705    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-api-access-99zml" (OuterVolumeSpecName: "kube-api-access-99zml") pod "8858622a-dba1-4f5d-b2eb-7f1e0c85513b" (UID: "8858622a-dba1-4f5d-b2eb-7f1e0c85513b"). InnerVolumeSpecName "kube-api-access-99zml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:04:26.148643 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.148612    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8858622a-dba1-4f5d-b2eb-7f1e0c85513b" (UID: "8858622a-dba1-4f5d-b2eb-7f1e0c85513b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:04:26.239720 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.239626    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:04:26.239720 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.239659    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99zml\" (UniqueName: \"kubernetes.io/projected/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-api-access-99zml\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:04:26.239720 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.239673    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:04:26.239720 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.239686    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8858622a-dba1-4f5d-b2eb-7f1e0c85513b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:04:26.905140 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.905108    2583 generic.go:356] "Generic (PLEG): container finished" podID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerID="946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282" exitCode=0
Sep 08 14:04:26.905312 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.905177    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"
Sep 08 14:04:26.905312 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.905193    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" event={"ID":"8858622a-dba1-4f5d-b2eb-7f1e0c85513b","Type":"ContainerDied","Data":"946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282"}
Sep 08 14:04:26.905312 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.905263    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd" event={"ID":"8858622a-dba1-4f5d-b2eb-7f1e0c85513b","Type":"ContainerDied","Data":"55d96758b0163b55eb3df0b9ac14bb3c4f96cc39795cb59e8c91dfaaeef0f059"}
Sep 08 14:04:26.905312 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.905303    2583 scope.go:117] "RemoveContainer" containerID="f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490"
Sep 08 14:04:26.914282 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.914254    2583 scope.go:117] "RemoveContainer" containerID="946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282"
Sep 08 14:04:26.922126 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.922107    2583 scope.go:117] "RemoveContainer" containerID="ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f"
Sep 08 14:04:26.929394 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.929370    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"]
Sep 08 14:04:26.930379 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.930359    2583 scope.go:117] "RemoveContainer" containerID="f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490"
Sep 08 14:04:26.931209 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:04:26.930903    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490\": container with ID starting with f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490 not found: ID does not exist" containerID="f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490"
Sep 08 14:04:26.931209 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.930981    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490"} err="failed to get container status \"f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490\": rpc error: code = NotFound desc = could not find container \"f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490\": container with ID starting with f3bf2f3f5402eafc425c356374e1b0addd931f8baef083a7900cf863dcf3b490 not found: ID does not exist"
Sep 08 14:04:26.931209 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.931006    2583 scope.go:117] "RemoveContainer" containerID="946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282"
Sep 08 14:04:26.931427 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:04:26.931375    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282\": container with ID starting with 946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282 not found: ID does not exist" containerID="946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282"
Sep 08 14:04:26.931427 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.931407    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282"} err="failed to get container status \"946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282\": rpc error: code = NotFound desc = could not find container \"946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282\": container with ID starting with 946c5b5bde8a222fa6543a0ee2f9689c148ed7665ba1ab391cce34328d0ce282 not found: ID does not exist"
Sep 08 14:04:26.931540 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.931430    2583 scope.go:117] "RemoveContainer" containerID="ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f"
Sep 08 14:04:26.931734 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:04:26.931716    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f\": container with ID starting with ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f not found: ID does not exist" containerID="ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f"
Sep 08 14:04:26.931788 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.931740    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f"} err="failed to get container status \"ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f\": rpc error: code = NotFound desc = could not find container \"ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f\": container with ID starting with ea2908362be81f51713e5b46259cb52ed4b895d81e69b5f2ecd2539af291044f not found: ID does not exist"
Sep 08 14:04:26.933052 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:26.933034    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-qkqbd"]
Sep 08 14:04:27.595327 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:27.595293    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" path="/var/lib/kubelet/pods/8858622a-dba1-4f5d-b2eb-7f1e0c85513b/volumes"
Sep 08 14:04:28.579242 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.579209    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"]
Sep 08 14:04:28.579627 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.579613    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="storage-initializer"
Sep 08 14:04:28.579674 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.579629    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="storage-initializer"
Sep 08 14:04:28.579674 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.579639    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kube-rbac-proxy"
Sep 08 14:04:28.579674 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.579648    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kube-rbac-proxy"
Sep 08 14:04:28.579674 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.579668    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container"
Sep 08 14:04:28.579674 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.579674    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container"
Sep 08 14:04:28.579834 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.579726    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kube-rbac-proxy"
Sep 08 14:04:28.579834 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.579738    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8858622a-dba1-4f5d-b2eb-7f1e0c85513b" containerName="kserve-container"
Sep 08 14:04:28.584812 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.584791    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.587102 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.587082    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 08 14:04:28.587207 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.587084    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 08 14:04:28.587419 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.587405    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 14:04:28.594856 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.594836    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"]
Sep 08 14:04:28.658680 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.658638    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c55qz\" (UniqueName: \"kubernetes.io/projected/4728dddf-0c66-401c-adcc-b159e30f1862-kube-api-access-c55qz\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.658680 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.658678    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4728dddf-0c66-401c-adcc-b159e30f1862-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.659159 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.658698    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.659159 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.658759    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4728dddf-0c66-401c-adcc-b159e30f1862-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.659159 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.658828    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.743414 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.743367    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:28.759860 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.759825    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c55qz\" (UniqueName: \"kubernetes.io/projected/4728dddf-0c66-401c-adcc-b159e30f1862-kube-api-access-c55qz\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.760055 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.759867    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4728dddf-0c66-401c-adcc-b159e30f1862-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.760055 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.759899    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.760055 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.759935    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4728dddf-0c66-401c-adcc-b159e30f1862-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.760241 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.760058    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.760241 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:04:28.760071    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 08 14:04:28.760241 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:04:28.760140    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4728dddf-0c66-401c-adcc-b159e30f1862-proxy-tls podName:4728dddf-0c66-401c-adcc-b159e30f1862 nodeName:}" failed. No retries permitted until 2026-09-08 14:04:29.260120543 +0000 UTC m=+2898.189024269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4728dddf-0c66-401c-adcc-b159e30f1862-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" (UID: "4728dddf-0c66-401c-adcc-b159e30f1862") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 08 14:04:28.760360 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.760307    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4728dddf-0c66-401c-adcc-b159e30f1862-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.760638 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.760621    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.760762 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.760742    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:28.768899 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:28.768881    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c55qz\" (UniqueName: \"kubernetes.io/projected/4728dddf-0c66-401c-adcc-b159e30f1862-kube-api-access-c55qz\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:29.026214 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:29.026136    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:04:29.263433 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:29.263401    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4728dddf-0c66-401c-adcc-b159e30f1862-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:29.266104 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:29.266079    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4728dddf-0c66-401c-adcc-b159e30f1862-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:29.496072 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:29.496038    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:29.770301 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:29.770217    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"]
Sep 08 14:04:29.773731 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:04:29.773702    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4728dddf_0c66_401c_adcc_b159e30f1862.slice/crio-10d39c0f1872d092a0b3c33b4ef15c725210376ee1211fe315a6f8112f3e1578 WatchSource:0}: Error finding container 10d39c0f1872d092a0b3c33b4ef15c725210376ee1211fe315a6f8112f3e1578: Status 404 returned error can't find the container with id 10d39c0f1872d092a0b3c33b4ef15c725210376ee1211fe315a6f8112f3e1578
Sep 08 14:04:29.921916 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:29.921881    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" event={"ID":"4728dddf-0c66-401c-adcc-b159e30f1862","Type":"ContainerStarted","Data":"cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1"}
Sep 08 14:04:29.922091 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:29.921938    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" event={"ID":"4728dddf-0c66-401c-adcc-b159e30f1862","Type":"ContainerStarted","Data":"10d39c0f1872d092a0b3c33b4ef15c725210376ee1211fe315a6f8112f3e1578"}
Sep 08 14:04:33.732927 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:33.732885    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 08 14:04:34.946872 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:34.946836    2583 generic.go:356] "Generic (PLEG): container finished" podID="4728dddf-0c66-401c-adcc-b159e30f1862" containerID="cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1" exitCode=0
Sep 08 14:04:34.947333 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:34.946892    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" event={"ID":"4728dddf-0c66-401c-adcc-b159e30f1862","Type":"ContainerDied","Data":"cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1"}
Sep 08 14:04:35.953388 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:35.953358    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" event={"ID":"4728dddf-0c66-401c-adcc-b159e30f1862","Type":"ContainerStarted","Data":"ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277"}
Sep 08 14:04:35.953760 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:35.953426    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" event={"ID":"4728dddf-0c66-401c-adcc-b159e30f1862","Type":"ContainerStarted","Data":"4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e"}
Sep 08 14:04:35.953760 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:35.953619    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:35.977071 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:35.977025    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" podStartSLOduration=3.6656273219999997 podStartE2EDuration="7.976997669s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.311370347s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:04:35.975415881 +0000 UTC m=+2904.904319640" watchObservedRunningTime="2026-09-08 14:04:35.976997669 +0000 UTC m=+2904.905901411"
Sep 08 14:04:36.958322 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:36.958285    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:36.959568 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:36.959540    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 08 14:04:37.962986 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:37.962951    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 08 14:04:42.967989 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:42.967961    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:04:42.968609 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:42.968579    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 08 14:04:43.732819 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:43.732777    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 08 14:04:52.968908 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:52.968865    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 08 14:04:53.733107 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:04:53.733061    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 08 14:05:02.968944 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:02.968902    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 08 14:05:03.733176 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:03.733148    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:05:12.968731 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:12.968688    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 08 14:05:13.803049 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:13.801615    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"]
Sep 08 14:05:13.803049 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:13.802075    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container" containerID="cri-o://b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544" gracePeriod=30
Sep 08 14:05:13.803049 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:13.802359    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kube-rbac-proxy" containerID="cri-o://24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf" gracePeriod=30
Sep 08 14:05:13.812592 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:13.812551    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:14.126826 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:14.126795    2583 generic.go:356] "Generic (PLEG): container finished" podID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerID="24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf" exitCode=2
Sep 08 14:05:14.127232 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:14.126872    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" event={"ID":"a42f9d47-41bb-4f06-85ae-ae0409e65e23","Type":"ContainerDied","Data":"24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf"}
Sep 08 14:05:14.919209 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:14.919172    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:15.254505 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:15.254419    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:18.741114 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.741090    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:05:18.798760 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.798689    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9btbg\" (UniqueName: \"kubernetes.io/projected/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-api-access-9btbg\") pod \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") "
Sep 08 14:05:18.798926 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.798755    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42f9d47-41bb-4f06-85ae-ae0409e65e23-proxy-tls\") pod \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") "
Sep 08 14:05:18.798926 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.798790    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kserve-provision-location\") pod \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") "
Sep 08 14:05:18.798926 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.798827    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-cabundle-cert\") pod \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") "
Sep 08 14:05:18.798926 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.798857    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-rbac-proxy-sar-config\") pod \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\" (UID: \"a42f9d47-41bb-4f06-85ae-ae0409e65e23\") "
Sep 08 14:05:18.799187 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.799103    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a42f9d47-41bb-4f06-85ae-ae0409e65e23" (UID: "a42f9d47-41bb-4f06-85ae-ae0409e65e23"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:05:18.799287 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.799262    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a42f9d47-41bb-4f06-85ae-ae0409e65e23" (UID: "a42f9d47-41bb-4f06-85ae-ae0409e65e23"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:05:18.799346 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.799318    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a42f9d47-41bb-4f06-85ae-ae0409e65e23" (UID: "a42f9d47-41bb-4f06-85ae-ae0409e65e23"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:05:18.800894 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.800877    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-api-access-9btbg" (OuterVolumeSpecName: "kube-api-access-9btbg") pod "a42f9d47-41bb-4f06-85ae-ae0409e65e23" (UID: "a42f9d47-41bb-4f06-85ae-ae0409e65e23"). InnerVolumeSpecName "kube-api-access-9btbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:05:18.801377 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.801355    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a42f9d47-41bb-4f06-85ae-ae0409e65e23-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a42f9d47-41bb-4f06-85ae-ae0409e65e23" (UID: "a42f9d47-41bb-4f06-85ae-ae0409e65e23"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:05:18.900264 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.900228    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:18.900264 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.900257    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9btbg\" (UniqueName: \"kubernetes.io/projected/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kube-api-access-9btbg\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:18.900264 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.900272    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a42f9d47-41bb-4f06-85ae-ae0409e65e23-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:18.900509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.900286    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a42f9d47-41bb-4f06-85ae-ae0409e65e23-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:18.900509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:18.900298    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a42f9d47-41bb-4f06-85ae-ae0409e65e23-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:19.152544 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.152507    2583 generic.go:356] "Generic (PLEG): container finished" podID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerID="b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544" exitCode=0
Sep 08 14:05:19.152746 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.152575    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"
Sep 08 14:05:19.152746 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.152589    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" event={"ID":"a42f9d47-41bb-4f06-85ae-ae0409e65e23","Type":"ContainerDied","Data":"b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544"}
Sep 08 14:05:19.152746 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.152648    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" event={"ID":"a42f9d47-41bb-4f06-85ae-ae0409e65e23","Type":"ContainerDied","Data":"e6096a532640807b545ef9ba44de6ac88f57c4571c7c396ec01d42bf63bfebb6"}
Sep 08 14:05:19.152746 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.152678    2583 scope.go:117] "RemoveContainer" containerID="24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf"
Sep 08 14:05:19.161879 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.161861    2583 scope.go:117] "RemoveContainer" containerID="b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544"
Sep 08 14:05:19.169515 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.169499    2583 scope.go:117] "RemoveContainer" containerID="908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f"
Sep 08 14:05:19.175202 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.175181    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"]
Sep 08 14:05:19.177469 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.177452    2583 scope.go:117] "RemoveContainer" containerID="24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf"
Sep 08 14:05:19.177752 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:05:19.177728    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf\": container with ID starting with 24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf not found: ID does not exist" containerID="24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf"
Sep 08 14:05:19.177846 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.177766    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf"} err="failed to get container status \"24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf\": rpc error: code = NotFound desc = could not find container \"24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf\": container with ID starting with 24978d7279f801d243a3f491e96b33bfb0e247b12942981260e1fc43d2b119cf not found: ID does not exist"
Sep 08 14:05:19.177846 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.177788    2583 scope.go:117] "RemoveContainer" containerID="b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544"
Sep 08 14:05:19.178067 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:05:19.178051    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544\": container with ID starting with b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544 not found: ID does not exist" containerID="b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544"
Sep 08 14:05:19.178120 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.178072    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544"} err="failed to get container status \"b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544\": rpc error: code = NotFound desc = could not find container \"b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544\": container with ID starting with b63322d92a7f9c86021613dd471d4390783f461c35607aa539cfc6851445a544 not found: ID does not exist"
Sep 08 14:05:19.178120 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.178088    2583 scope.go:117] "RemoveContainer" containerID="908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f"
Sep 08 14:05:19.178344 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:05:19.178313    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f\": container with ID starting with 908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f not found: ID does not exist" containerID="908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f"
Sep 08 14:05:19.178403 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.178341    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f"} err="failed to get container status \"908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f\": rpc error: code = NotFound desc = could not find container \"908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f\": container with ID starting with 908f9d9d5044bef2a232fd1f874eceef7ebb75b5528cdc34e7ade1e9cfccbb7f not found: ID does not exist"
Sep 08 14:05:19.181444 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.181425    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn"]
Sep 08 14:05:19.257466 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.257435    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:19.595253 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.595218    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" path="/var/lib/kubelet/pods/a42f9d47-41bb-4f06-85ae-ae0409e65e23/volumes"
Sep 08 14:05:19.729264 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:19.729224    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-588548cf85-brlcn" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 08 14:05:22.969173 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:22.969141    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:05:24.987552 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:24.987519    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:26.006638 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.006601    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:26.033329 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.033292    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"]
Sep 08 14:05:26.033644 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.033631    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="storage-initializer"
Sep 08 14:05:26.033698 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.033646    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="storage-initializer"
Sep 08 14:05:26.033698 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.033656    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container"
Sep 08 14:05:26.033698 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.033662    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container"
Sep 08 14:05:26.033698 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.033682    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kube-rbac-proxy"
Sep 08 14:05:26.033698 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.033687    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kube-rbac-proxy"
Sep 08 14:05:26.033869 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.033742    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kube-rbac-proxy"
Sep 08 14:05:26.033869 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.033754    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="a42f9d47-41bb-4f06-85ae-ae0409e65e23" containerName="kserve-container"
Sep 08 14:05:26.036983 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.036962    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.038910 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.038888    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 08 14:05:26.039047 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.038949    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 08 14:05:26.039282 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.039266    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 08 14:05:26.050663 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.050642    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"]
Sep 08 14:05:26.061865 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.061834    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.062002 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.061914    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.062002 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.061947    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c02b8e7-6b71-4684-8f66-8f4ed3792621-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.062111 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.062043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.062111 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.062081    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qdd4\" (UniqueName: \"kubernetes.io/projected/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-api-access-7qdd4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.162530 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.162491    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c02b8e7-6b71-4684-8f66-8f4ed3792621-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.162710 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.162539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.162710 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.162584    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qdd4\" (UniqueName: \"kubernetes.io/projected/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-api-access-7qdd4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.162710 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.162647    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.162710 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:05:26.162653    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 08 14:05:26.162919 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:05:26.162730    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c02b8e7-6b71-4684-8f66-8f4ed3792621-proxy-tls podName:5c02b8e7-6b71-4684-8f66-8f4ed3792621 nodeName:}" failed. No retries permitted until 2026-09-08 14:05:26.662708201 +0000 UTC m=+2955.591611925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5c02b8e7-6b71-4684-8f66-8f4ed3792621-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" (UID: "5c02b8e7-6b71-4684-8f66-8f4ed3792621") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Sep 08 14:05:26.162919 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.162863    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.163143 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.163121    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.163343 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.163325    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.163425 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.163409    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.172940 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.172916    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qdd4\" (UniqueName: \"kubernetes.io/projected/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-api-access-7qdd4\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.386835 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.386799    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:26.666086 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.665975    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c02b8e7-6b71-4684-8f66-8f4ed3792621-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.668616 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.668586    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c02b8e7-6b71-4684-8f66-8f4ed3792621-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:26.948202 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:26.948098    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:27.225619 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:27.225597    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"]
Sep 08 14:05:27.228113 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:05:27.228087    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c02b8e7_6b71_4684_8f66_8f4ed3792621.slice/crio-894c03a8945e21a0855ea1b1b81d9fb081a0b1c8afe7327d87184633de14843d WatchSource:0}: Error finding container 894c03a8945e21a0855ea1b1b81d9fb081a0b1c8afe7327d87184633de14843d: Status 404 returned error can't find the container with id 894c03a8945e21a0855ea1b1b81d9fb081a0b1c8afe7327d87184633de14843d
Sep 08 14:05:28.205084 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:28.205051    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" event={"ID":"5c02b8e7-6b71-4684-8f66-8f4ed3792621","Type":"ContainerStarted","Data":"8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb"}
Sep 08 14:05:28.205282 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:28.205145    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" event={"ID":"5c02b8e7-6b71-4684-8f66-8f4ed3792621","Type":"ContainerStarted","Data":"894c03a8945e21a0855ea1b1b81d9fb081a0b1c8afe7327d87184633de14843d"}
Sep 08 14:05:29.210957 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:29.210922    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerID="8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb" exitCode=0
Sep 08 14:05:29.211468 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:29.210994    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" event={"ID":"5c02b8e7-6b71-4684-8f66-8f4ed3792621","Type":"ContainerDied","Data":"8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb"}
Sep 08 14:05:29.951438 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:29.951401    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"]
Sep 08 14:05:29.951659 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:29.951633    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container" containerID="cri-o://4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e" gracePeriod=30
Sep 08 14:05:29.951749 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:29.951674    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kube-rbac-proxy" containerID="cri-o://ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277" gracePeriod=30
Sep 08 14:05:29.972527 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:29.972485    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:30.218321 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:30.218222    2583 generic.go:356] "Generic (PLEG): container finished" podID="4728dddf-0c66-401c-adcc-b159e30f1862" containerID="ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277" exitCode=2
Sep 08 14:05:30.218788 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:30.218311    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" event={"ID":"4728dddf-0c66-401c-adcc-b159e30f1862","Type":"ContainerDied","Data":"ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277"}
Sep 08 14:05:30.220654 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:30.220628    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" event={"ID":"5c02b8e7-6b71-4684-8f66-8f4ed3792621","Type":"ContainerStarted","Data":"d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808"}
Sep 08 14:05:30.220769 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:30.220684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" event={"ID":"5c02b8e7-6b71-4684-8f66-8f4ed3792621","Type":"ContainerStarted","Data":"ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390"}
Sep 08 14:05:30.220845 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:30.220802    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:30.242780 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:30.242730    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podStartSLOduration=3.184710845 podStartE2EDuration="4.242715232s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.058004387s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:05:30.240912692 +0000 UTC m=+2959.169816433" watchObservedRunningTime="2026-09-08 14:05:30.242715232 +0000 UTC m=+2959.171618973"
Sep 08 14:05:31.226367 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:31.226331    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:31.227777 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:31.227752    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:05:32.231121 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.231083    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:05:32.701902 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.701880    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:05:32.820552 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.820524    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-cabundle-cert\") pod \"4728dddf-0c66-401c-adcc-b159e30f1862\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") "
Sep 08 14:05:32.820720 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.820583    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c55qz\" (UniqueName: \"kubernetes.io/projected/4728dddf-0c66-401c-adcc-b159e30f1862-kube-api-access-c55qz\") pod \"4728dddf-0c66-401c-adcc-b159e30f1862\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") "
Sep 08 14:05:32.820720 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.820604    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-kube-rbac-proxy-sar-config\") pod \"4728dddf-0c66-401c-adcc-b159e30f1862\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") "
Sep 08 14:05:32.820816 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.820721    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4728dddf-0c66-401c-adcc-b159e30f1862-kserve-provision-location\") pod \"4728dddf-0c66-401c-adcc-b159e30f1862\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") "
Sep 08 14:05:32.820816 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.820780    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4728dddf-0c66-401c-adcc-b159e30f1862-proxy-tls\") pod \"4728dddf-0c66-401c-adcc-b159e30f1862\" (UID: \"4728dddf-0c66-401c-adcc-b159e30f1862\") "
Sep 08 14:05:32.820982 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.820958    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4728dddf-0c66-401c-adcc-b159e30f1862" (UID: "4728dddf-0c66-401c-adcc-b159e30f1862"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:05:32.821053 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.820977    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4728dddf-0c66-401c-adcc-b159e30f1862" (UID: "4728dddf-0c66-401c-adcc-b159e30f1862"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:05:32.821153 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.821137    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:32.821196 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.821158    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4728dddf-0c66-401c-adcc-b159e30f1862-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:32.822926 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.822904    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4728dddf-0c66-401c-adcc-b159e30f1862-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4728dddf-0c66-401c-adcc-b159e30f1862" (UID: "4728dddf-0c66-401c-adcc-b159e30f1862"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:05:32.823112 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.823096    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4728dddf-0c66-401c-adcc-b159e30f1862-kube-api-access-c55qz" (OuterVolumeSpecName: "kube-api-access-c55qz") pod "4728dddf-0c66-401c-adcc-b159e30f1862" (UID: "4728dddf-0c66-401c-adcc-b159e30f1862"). InnerVolumeSpecName "kube-api-access-c55qz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:05:32.830146 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.830122    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4728dddf-0c66-401c-adcc-b159e30f1862-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4728dddf-0c66-401c-adcc-b159e30f1862" (UID: "4728dddf-0c66-401c-adcc-b159e30f1862"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:05:32.921976 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.921938    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4728dddf-0c66-401c-adcc-b159e30f1862-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:32.921976 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.921969    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c55qz\" (UniqueName: \"kubernetes.io/projected/4728dddf-0c66-401c-adcc-b159e30f1862-kube-api-access-c55qz\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:32.921976 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:32.921979    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4728dddf-0c66-401c-adcc-b159e30f1862-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:05:33.238698 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.238614    2583 generic.go:356] "Generic (PLEG): container finished" podID="4728dddf-0c66-401c-adcc-b159e30f1862" containerID="4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e" exitCode=0
Sep 08 14:05:33.238698 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.238676    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"
Sep 08 14:05:33.239155 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.238707    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" event={"ID":"4728dddf-0c66-401c-adcc-b159e30f1862","Type":"ContainerDied","Data":"4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e"}
Sep 08 14:05:33.239155 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.238779    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6" event={"ID":"4728dddf-0c66-401c-adcc-b159e30f1862","Type":"ContainerDied","Data":"10d39c0f1872d092a0b3c33b4ef15c725210376ee1211fe315a6f8112f3e1578"}
Sep 08 14:05:33.239155 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.238819    2583 scope.go:117] "RemoveContainer" containerID="ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277"
Sep 08 14:05:33.249445 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.249402    2583 scope.go:117] "RemoveContainer" containerID="4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e"
Sep 08 14:05:33.259093 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.259069    2583 scope.go:117] "RemoveContainer" containerID="cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1"
Sep 08 14:05:33.267262 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.267239    2583 scope.go:117] "RemoveContainer" containerID="ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277"
Sep 08 14:05:33.267563 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:05:33.267541    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277\": container with ID starting with ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277 not found: ID does not exist" containerID="ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277"
Sep 08 14:05:33.267680 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.267572    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277"} err="failed to get container status \"ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277\": rpc error: code = NotFound desc = could not find container \"ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277\": container with ID starting with ba9ba55cb84c3bb3f1ff95dccbc4ba4d52fc8be36bef32f0a0c85e13eebc3277 not found: ID does not exist"
Sep 08 14:05:33.267680 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.267590    2583 scope.go:117] "RemoveContainer" containerID="4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e"
Sep 08 14:05:33.267680 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.267595    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"]
Sep 08 14:05:33.267819 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:05:33.267805    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e\": container with ID starting with 4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e not found: ID does not exist" containerID="4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e"
Sep 08 14:05:33.267853 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.267826    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e"} err="failed to get container status \"4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e\": rpc error: code = NotFound desc = could not find container \"4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e\": container with ID starting with 4e0029c4d585e62a0aee8c07b056e6a7e53dbd01145f899dda40e89fc554667e not found: ID does not exist"
Sep 08 14:05:33.267853 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.267842    2583 scope.go:117] "RemoveContainer" containerID="cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1"
Sep 08 14:05:33.268085 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:05:33.268071    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1\": container with ID starting with cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1 not found: ID does not exist" containerID="cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1"
Sep 08 14:05:33.268135 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.268088    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1"} err="failed to get container status \"cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1\": rpc error: code = NotFound desc = could not find container \"cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1\": container with ID starting with cd133265faa9e3519e122d6d07ecf4ed8c9b2a266d2ee2b2a9f9a8f14a73c9f1 not found: ID does not exist"
Sep 08 14:05:33.272068 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.272044    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-k2sc6"]
Sep 08 14:05:33.596549 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:33.596514    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" path="/var/lib/kubelet/pods/4728dddf-0c66-401c-adcc-b159e30f1862/volumes"
Sep 08 14:05:35.126722 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.126683    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:35.154148 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.154102    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"]
Sep 08 14:05:35.154638 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.154619    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="storage-initializer"
Sep 08 14:05:35.154638 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.154638    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="storage-initializer"
Sep 08 14:05:35.154846 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.154666    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container"
Sep 08 14:05:35.154846 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.154674    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container"
Sep 08 14:05:35.154846 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.154683    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kube-rbac-proxy"
Sep 08 14:05:35.154846 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.154691    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kube-rbac-proxy"
Sep 08 14:05:35.154846 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.154772    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kserve-container"
Sep 08 14:05:35.154846 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.154782    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4728dddf-0c66-401c-adcc-b159e30f1862" containerName="kube-rbac-proxy"
Sep 08 14:05:35.160265 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.160244    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.162568 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.162544    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 08 14:05:35.162673 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.162598    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 14:05:35.162734 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.162554    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 08 14:05:35.167850 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.167824    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"]
Sep 08 14:05:35.240026 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.239973    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3239708f-0b72-45fd-bcb8-80958abab35a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.240199 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.240050    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.240199 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.240110    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3239708f-0b72-45fd-bcb8-80958abab35a-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.240276 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.240198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.240276 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.240227    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5jmf\" (UniqueName: \"kubernetes.io/projected/3239708f-0b72-45fd-bcb8-80958abab35a-kube-api-access-n5jmf\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.243649 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.243610    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:05:35.341693 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.341660    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3239708f-0b72-45fd-bcb8-80958abab35a-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.341885 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.341713    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.341885 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.341741    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5jmf\" (UniqueName: \"kubernetes.io/projected/3239708f-0b72-45fd-bcb8-80958abab35a-kube-api-access-n5jmf\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.341885 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.341786    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3239708f-0b72-45fd-bcb8-80958abab35a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.341885 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.341821    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.342228 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.342209    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3239708f-0b72-45fd-bcb8-80958abab35a-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.342596 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.342569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.342699 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.342569    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.344359 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.344339    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3239708f-0b72-45fd-bcb8-80958abab35a-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.350034 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.349997    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5jmf\" (UniqueName: \"kubernetes.io/projected/3239708f-0b72-45fd-bcb8-80958abab35a-kube-api-access-n5jmf\") pod \"isvc-pmml-predictor-8674795f59-njfc5\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.473725 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.473623    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:35.748551 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:35.748524    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"]
Sep 08 14:05:35.751245 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:05:35.751221    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3239708f_0b72_45fd_bcb8_80958abab35a.slice/crio-206e65496f8bb04aeb193fd714e1cbb9ad3793dbfdd30170986b60a1b01d5b8a WatchSource:0}: Error finding container 206e65496f8bb04aeb193fd714e1cbb9ad3793dbfdd30170986b60a1b01d5b8a: Status 404 returned error can't find the container with id 206e65496f8bb04aeb193fd714e1cbb9ad3793dbfdd30170986b60a1b01d5b8a
Sep 08 14:05:36.255293 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:36.255246    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" event={"ID":"3239708f-0b72-45fd-bcb8-80958abab35a","Type":"ContainerStarted","Data":"238d1fb7bf8c3c77bab88efac37a26fa210d75d855b329570d9767f088a5974d"}
Sep 08 14:05:36.255697 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:36.255321    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" event={"ID":"3239708f-0b72-45fd-bcb8-80958abab35a","Type":"ContainerStarted","Data":"206e65496f8bb04aeb193fd714e1cbb9ad3793dbfdd30170986b60a1b01d5b8a"}
Sep 08 14:05:37.235997 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:37.235965    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:05:37.236482 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:37.236459    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:05:40.275835 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:40.275803    2583 generic.go:356] "Generic (PLEG): container finished" podID="3239708f-0b72-45fd-bcb8-80958abab35a" containerID="238d1fb7bf8c3c77bab88efac37a26fa210d75d855b329570d9767f088a5974d" exitCode=0
Sep 08 14:05:40.276324 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:40.275891    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" event={"ID":"3239708f-0b72-45fd-bcb8-80958abab35a","Type":"ContainerDied","Data":"238d1fb7bf8c3c77bab88efac37a26fa210d75d855b329570d9767f088a5974d"}
Sep 08 14:05:40.277264 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:40.277250    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 14:05:47.236571 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:47.236530    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:05:50.343482 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:50.343447    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" event={"ID":"3239708f-0b72-45fd-bcb8-80958abab35a","Type":"ContainerStarted","Data":"2e8bcbb0413f834b2b1f59121b73c811d8dad454db3cbd162d21144b9780e433"}
Sep 08 14:05:50.343873 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:50.343512    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" event={"ID":"3239708f-0b72-45fd-bcb8-80958abab35a","Type":"ContainerStarted","Data":"6fae3d9d8877b805db1a071c729b59eb42e4c86328efca6f750b105ee4805a2f"}
Sep 08 14:05:50.343873 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:50.343757    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:50.365684 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:50.365613    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podStartSLOduration=2.056177522 podStartE2EDuration="15.365592614s" totalImagesPullingTime="9.700024153s" totalInitContainerRuntime="3.609390939s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:05:50.36523007 +0000 UTC m=+2979.294133849" watchObservedRunningTime="2026-09-08 14:05:50.365592614 +0000 UTC m=+2979.294496358"
Sep 08 14:05:51.347685 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:51.347650    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:51.348935 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:51.348910    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:05:52.352620 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:52.352581    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:05:57.237418 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:57.237373    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:05:57.356768 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:57.356738    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:05:57.357261 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:05:57.357237    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:06:07.237418 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:07.237381    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:06:07.357923 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:07.357882    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:06:11.779083 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:11.779054    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 14:06:11.785798 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:11.785776    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 14:06:17.236561 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:17.236521    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:06:17.357380 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:17.357343    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:06:27.237283 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:27.237236    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:06:27.357702 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:27.357661    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:06:37.237111 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:37.237071    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:06:37.357818 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:37.357776    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:06:42.248830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:42.248793    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:47.237979 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:47.237951    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:06:47.357477 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:47.357435    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:06:56.071161 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:56.071120    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"]
Sep 08 14:06:56.071601 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:56.071387    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" containerID="cri-o://ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390" gracePeriod=30
Sep 08 14:06:56.071601 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:56.071471    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kube-rbac-proxy" containerID="cri-o://d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808" gracePeriod=30
Sep 08 14:06:56.096077 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:56.096042    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:56.638218 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:56.638179    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerID="d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808" exitCode=2
Sep 08 14:06:56.638390 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:56.638247    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" event={"ID":"5c02b8e7-6b71-4684-8f66-8f4ed3792621","Type":"ContainerDied","Data":"d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808"}
Sep 08 14:06:57.170752 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:57.170719    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:06:57.232163 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:57.232117    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 08 14:06:57.236633 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:57.236583    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 08 14:06:57.358140 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:57.358092    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:06:57.523753 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:06:57.523655    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:01.118204 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.118175    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:07:01.208944 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.208845    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qdd4\" (UniqueName: \"kubernetes.io/projected/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-api-access-7qdd4\") pod \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") "
Sep 08 14:07:01.208944 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.208895    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kserve-provision-location\") pod \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") "
Sep 08 14:07:01.208944 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.208929    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-rbac-proxy-sar-config\") pod \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") "
Sep 08 14:07:01.209283 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.208951    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-cabundle-cert\") pod \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") "
Sep 08 14:07:01.209283 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.208982    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c02b8e7-6b71-4684-8f66-8f4ed3792621-proxy-tls\") pod \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\" (UID: \"5c02b8e7-6b71-4684-8f66-8f4ed3792621\") "
Sep 08 14:07:01.209378 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.209316    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c02b8e7-6b71-4684-8f66-8f4ed3792621" (UID: "5c02b8e7-6b71-4684-8f66-8f4ed3792621"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:07:01.209378 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.209344    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c02b8e7-6b71-4684-8f66-8f4ed3792621" (UID: "5c02b8e7-6b71-4684-8f66-8f4ed3792621"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:07:01.209378 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.209363    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c02b8e7-6b71-4684-8f66-8f4ed3792621" (UID: "5c02b8e7-6b71-4684-8f66-8f4ed3792621"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:07:01.211364 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.211326    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c02b8e7-6b71-4684-8f66-8f4ed3792621-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c02b8e7-6b71-4684-8f66-8f4ed3792621" (UID: "5c02b8e7-6b71-4684-8f66-8f4ed3792621"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:07:01.211364 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.211357    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-api-access-7qdd4" (OuterVolumeSpecName: "kube-api-access-7qdd4") pod "5c02b8e7-6b71-4684-8f66-8f4ed3792621" (UID: "5c02b8e7-6b71-4684-8f66-8f4ed3792621"). InnerVolumeSpecName "kube-api-access-7qdd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:07:01.309874 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.309833    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:01.309874 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.309872    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c02b8e7-6b71-4684-8f66-8f4ed3792621-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:01.309874 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.309882    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c02b8e7-6b71-4684-8f66-8f4ed3792621-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:01.310158 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.309893    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qdd4\" (UniqueName: \"kubernetes.io/projected/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kube-api-access-7qdd4\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:01.310158 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.309902    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c02b8e7-6b71-4684-8f66-8f4ed3792621-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:01.661607 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.661574    2583 generic.go:356] "Generic (PLEG): container finished" podID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerID="ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390" exitCode=0
Sep 08 14:07:01.661800 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.661627    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" event={"ID":"5c02b8e7-6b71-4684-8f66-8f4ed3792621","Type":"ContainerDied","Data":"ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390"}
Sep 08 14:07:01.667509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.662023    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc" event={"ID":"5c02b8e7-6b71-4684-8f66-8f4ed3792621","Type":"ContainerDied","Data":"894c03a8945e21a0855ea1b1b81d9fb081a0b1c8afe7327d87184633de14843d"}
Sep 08 14:07:01.667509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.661630    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"
Sep 08 14:07:01.667509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.662200    2583 scope.go:117] "RemoveContainer" containerID="d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808"
Sep 08 14:07:01.676643 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.676623    2583 scope.go:117] "RemoveContainer" containerID="ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390"
Sep 08 14:07:01.683724 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.683693    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"]
Sep 08 14:07:01.686271 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.686250    2583 scope.go:117] "RemoveContainer" containerID="8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb"
Sep 08 14:07:01.687628 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.687607    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-76ff6cf8bf-r5ndc"]
Sep 08 14:07:01.696434 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.696390    2583 scope.go:117] "RemoveContainer" containerID="d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808"
Sep 08 14:07:01.696854 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:07:01.696828    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808\": container with ID starting with d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808 not found: ID does not exist" containerID="d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808"
Sep 08 14:07:01.696932 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.696868    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808"} err="failed to get container status \"d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808\": rpc error: code = NotFound desc = could not find container \"d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808\": container with ID starting with d9439186aa907eff2cdb3832ca567dca5e6ad4cf047719266dc85aa4033c7808 not found: ID does not exist"
Sep 08 14:07:01.696932 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.696896    2583 scope.go:117] "RemoveContainer" containerID="ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390"
Sep 08 14:07:01.697286 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:07:01.697262    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390\": container with ID starting with ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390 not found: ID does not exist" containerID="ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390"
Sep 08 14:07:01.697345 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.697293    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390"} err="failed to get container status \"ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390\": rpc error: code = NotFound desc = could not find container \"ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390\": container with ID starting with ae60db82dae7afd874bf9a18f055fbc671424c5e3856382cd53b4744cd009390 not found: ID does not exist"
Sep 08 14:07:01.697345 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.697310    2583 scope.go:117] "RemoveContainer" containerID="8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb"
Sep 08 14:07:01.697585 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:07:01.697566    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb\": container with ID starting with 8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb not found: ID does not exist" containerID="8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb"
Sep 08 14:07:01.697644 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:01.697588    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb"} err="failed to get container status \"8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb\": rpc error: code = NotFound desc = could not find container \"8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb\": container with ID starting with 8a377b3d0b1e1ed56442d8da822822ffa17f0bf7919eb5b07e4a451768d7b6fb not found: ID does not exist"
Sep 08 14:07:03.595758 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:03.595675    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" path="/var/lib/kubelet/pods/5c02b8e7-6b71-4684-8f66-8f4ed3792621/volumes"
Sep 08 14:07:06.590390 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:06.590350    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:07:07.201714 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:07.201677    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:08.758528 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:08.758492    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:16.591197 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:16.591167    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:07:26.254663 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:26.254629    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"]
Sep 08 14:07:26.257043 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:26.254891    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" containerID="cri-o://6fae3d9d8877b805db1a071c729b59eb42e4c86328efca6f750b105ee4805a2f" gracePeriod=30
Sep 08 14:07:26.257043 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:26.254919    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kube-rbac-proxy" containerID="cri-o://2e8bcbb0413f834b2b1f59121b73c811d8dad454db3cbd162d21144b9780e433" gracePeriod=30
Sep 08 14:07:26.277119 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:26.277084    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:26.590642 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:26.590608    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 08 14:07:26.772147 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:26.772113    2583 generic.go:356] "Generic (PLEG): container finished" podID="3239708f-0b72-45fd-bcb8-80958abab35a" containerID="2e8bcbb0413f834b2b1f59121b73c811d8dad454db3cbd162d21144b9780e433" exitCode=2
Sep 08 14:07:26.772147 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:26.772150    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" event={"ID":"3239708f-0b72-45fd-bcb8-80958abab35a","Type":"ContainerDied","Data":"2e8bcbb0413f834b2b1f59121b73c811d8dad454db3cbd162d21144b9780e433"}
Sep 08 14:07:27.352897 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:27.352850    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 08 14:07:29.787354 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.787315    2583 generic.go:356] "Generic (PLEG): container finished" podID="3239708f-0b72-45fd-bcb8-80958abab35a" containerID="6fae3d9d8877b805db1a071c729b59eb42e4c86328efca6f750b105ee4805a2f" exitCode=0
Sep 08 14:07:29.787830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.787381    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" event={"ID":"3239708f-0b72-45fd-bcb8-80958abab35a","Type":"ContainerDied","Data":"6fae3d9d8877b805db1a071c729b59eb42e4c86328efca6f750b105ee4805a2f"}
Sep 08 14:07:29.838164 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.838142    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:07:29.957564 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.957472    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3239708f-0b72-45fd-bcb8-80958abab35a-proxy-tls\") pod \"3239708f-0b72-45fd-bcb8-80958abab35a\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") "
Sep 08 14:07:29.957564 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.957517    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-cabundle-cert\") pod \"3239708f-0b72-45fd-bcb8-80958abab35a\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") "
Sep 08 14:07:29.957564 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.957556    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-kube-rbac-proxy-sar-config\") pod \"3239708f-0b72-45fd-bcb8-80958abab35a\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") "
Sep 08 14:07:29.957814 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.957625    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3239708f-0b72-45fd-bcb8-80958abab35a-kserve-provision-location\") pod \"3239708f-0b72-45fd-bcb8-80958abab35a\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") "
Sep 08 14:07:29.957814 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.957649    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5jmf\" (UniqueName: \"kubernetes.io/projected/3239708f-0b72-45fd-bcb8-80958abab35a-kube-api-access-n5jmf\") pod \"3239708f-0b72-45fd-bcb8-80958abab35a\" (UID: \"3239708f-0b72-45fd-bcb8-80958abab35a\") "
Sep 08 14:07:29.957973 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.957935    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3239708f-0b72-45fd-bcb8-80958abab35a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3239708f-0b72-45fd-bcb8-80958abab35a" (UID: "3239708f-0b72-45fd-bcb8-80958abab35a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:07:29.957973 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.957967    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3239708f-0b72-45fd-bcb8-80958abab35a" (UID: "3239708f-0b72-45fd-bcb8-80958abab35a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:07:29.958151 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.957982    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3239708f-0b72-45fd-bcb8-80958abab35a" (UID: "3239708f-0b72-45fd-bcb8-80958abab35a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:07:29.959763 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.959742    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3239708f-0b72-45fd-bcb8-80958abab35a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3239708f-0b72-45fd-bcb8-80958abab35a" (UID: "3239708f-0b72-45fd-bcb8-80958abab35a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:07:29.959896 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:29.959875    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3239708f-0b72-45fd-bcb8-80958abab35a-kube-api-access-n5jmf" (OuterVolumeSpecName: "kube-api-access-n5jmf") pod "3239708f-0b72-45fd-bcb8-80958abab35a" (UID: "3239708f-0b72-45fd-bcb8-80958abab35a"). InnerVolumeSpecName "kube-api-access-n5jmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:07:30.058393 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.058351    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3239708f-0b72-45fd-bcb8-80958abab35a-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:30.058393 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.058386    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5jmf\" (UniqueName: \"kubernetes.io/projected/3239708f-0b72-45fd-bcb8-80958abab35a-kube-api-access-n5jmf\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:30.058393 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.058399    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3239708f-0b72-45fd-bcb8-80958abab35a-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:30.058636 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.058411    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:30.058636 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.058422    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3239708f-0b72-45fd-bcb8-80958abab35a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:07:30.793453 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.793425    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"
Sep 08 14:07:30.793453 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.793441    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5" event={"ID":"3239708f-0b72-45fd-bcb8-80958abab35a","Type":"ContainerDied","Data":"206e65496f8bb04aeb193fd714e1cbb9ad3793dbfdd30170986b60a1b01d5b8a"}
Sep 08 14:07:30.793941 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.793511    2583 scope.go:117] "RemoveContainer" containerID="2e8bcbb0413f834b2b1f59121b73c811d8dad454db3cbd162d21144b9780e433"
Sep 08 14:07:30.802426 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.802410    2583 scope.go:117] "RemoveContainer" containerID="6fae3d9d8877b805db1a071c729b59eb42e4c86328efca6f750b105ee4805a2f"
Sep 08 14:07:30.809805 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.809786    2583 scope.go:117] "RemoveContainer" containerID="238d1fb7bf8c3c77bab88efac37a26fa210d75d855b329570d9767f088a5974d"
Sep 08 14:07:30.814653 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.814628    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"]
Sep 08 14:07:30.818636 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:30.818615    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-njfc5"]
Sep 08 14:07:31.456607 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.456564    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:31.456851 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.456610    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"]
Sep 08 14:07:31.457214 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457198    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="storage-initializer"
Sep 08 14:07:31.457214 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457216    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="storage-initializer"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457226    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457233    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457242    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457249    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457255    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kube-rbac-proxy"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457260    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kube-rbac-proxy"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457273    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="storage-initializer"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457279    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="storage-initializer"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457286    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kube-rbac-proxy"
Sep 08 14:07:31.457325 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457291    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kube-rbac-proxy"
Sep 08 14:07:31.457628 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457352    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kube-rbac-proxy"
Sep 08 14:07:31.457628 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457359    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kserve-container"
Sep 08 14:07:31.457628 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457365    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" containerName="kube-rbac-proxy"
Sep 08 14:07:31.457628 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.457373    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c02b8e7-6b71-4684-8f66-8f4ed3792621" containerName="kserve-container"
Sep 08 14:07:31.462247 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.462226    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.469480 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.469458    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 14:07:31.469598 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.469462    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:07:31.469598 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.469462    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 08 14:07:31.469598 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.469462    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 08 14:07:31.469598 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.469507    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 14:07:31.469813 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.469799    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 14:07:31.470149 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.470133    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 14:07:31.475554 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.475531    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"]
Sep 08 14:07:31.570827 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.570787    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2fc63ac-afab-4892-a059-aa5b96a80a71-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.571120 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.570851    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.571120 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.570919    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.571120 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.570953    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2fc63ac-afab-4892-a059-aa5b96a80a71-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.571120 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.570995    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-api-access-vb25b\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.595422 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.595388    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3239708f-0b72-45fd-bcb8-80958abab35a" path="/var/lib/kubelet/pods/3239708f-0b72-45fd-bcb8-80958abab35a/volumes"
Sep 08 14:07:31.672296 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.672257    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.672468 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.672309    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2fc63ac-afab-4892-a059-aa5b96a80a71-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.672468 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.672370    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-api-access-vb25b\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.672468 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.672420    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2fc63ac-afab-4892-a059-aa5b96a80a71-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.672635 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.672499    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.672790 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.672749    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2fc63ac-afab-4892-a059-aa5b96a80a71-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.673169 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.673146    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.673291 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.673276    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.675029 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.674991    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2fc63ac-afab-4892-a059-aa5b96a80a71-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.682793 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.682760    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-api-access-vb25b\") pod \"isvc-pmml-runtime-predictor-885fc8777-p8c79\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.772551 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.772451    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:31.866546 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:31.866514    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:07:32.051516 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:32.051492    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"]
Sep 08 14:07:32.053307 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:07:32.053270    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fc63ac_afab_4892_a059_aa5b96a80a71.slice/crio-a38305f9274ea8b4f6f9485ff6ce670f00f556603646239887a6165ed66e8ddf WatchSource:0}: Error finding container a38305f9274ea8b4f6f9485ff6ce670f00f556603646239887a6165ed66e8ddf: Status 404 returned error can't find the container with id a38305f9274ea8b4f6f9485ff6ce670f00f556603646239887a6165ed66e8ddf
Sep 08 14:07:32.804732 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:32.804684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" event={"ID":"d2fc63ac-afab-4892-a059-aa5b96a80a71","Type":"ContainerStarted","Data":"4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4"}
Sep 08 14:07:32.804913 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:32.804793    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" event={"ID":"d2fc63ac-afab-4892-a059-aa5b96a80a71","Type":"ContainerStarted","Data":"a38305f9274ea8b4f6f9485ff6ce670f00f556603646239887a6165ed66e8ddf"}
Sep 08 14:07:36.824947 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:36.824893    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerID="4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4" exitCode=0
Sep 08 14:07:36.825416 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:36.824961    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" event={"ID":"d2fc63ac-afab-4892-a059-aa5b96a80a71","Type":"ContainerDied","Data":"4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4"}
Sep 08 14:07:37.832039 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:37.831985    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" event={"ID":"d2fc63ac-afab-4892-a059-aa5b96a80a71","Type":"ContainerStarted","Data":"d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766"}
Sep 08 14:07:37.832434 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:37.832072    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" event={"ID":"d2fc63ac-afab-4892-a059-aa5b96a80a71","Type":"ContainerStarted","Data":"f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b"}
Sep 08 14:07:37.832434 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:37.832256    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:37.856922 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:37.856868    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podStartSLOduration=2.884240589 podStartE2EDuration="6.856853832s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.972613243s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:07:37.853912061 +0000 UTC m=+3086.782815843" watchObservedRunningTime="2026-09-08 14:07:37.856853832 +0000 UTC m=+3086.785757608"
Sep 08 14:07:38.837867 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:38.837839    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:38.839144 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:38.839122    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:07:39.841661 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:39.841618    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:07:44.846032 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:44.845982    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:07:44.846553 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:44.846525    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:07:54.846721 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:07:54.846682    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:08:00.166770 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:00.166722    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:04.846604 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:04.846555    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:08:14.846869 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:14.846830    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:08:24.847147 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:24.847100    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:08:28.319891 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:28.319837    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:29.350040 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:29.349979    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:29.625370 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:29.625291    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:34.846823 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:34.846736    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:08:39.361887 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:39.361841    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:40.489158 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:40.489124    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:40.728740 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:40.728709    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:08:44.846666 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:44.846626    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:08:54.846914 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:08:54.846867    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:09:04.847869 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:04.847838    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:09:12.936992 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:12.936957    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"]
Sep 08 14:09:12.937525 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:12.937238    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" containerID="cri-o://f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b" gracePeriod=30
Sep 08 14:09:12.937525 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:12.937372    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kube-rbac-proxy" containerID="cri-o://d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766" gracePeriod=30
Sep 08 14:09:13.258880 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:13.258767    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerID="d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766" exitCode=2
Sep 08 14:09:13.258880 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:13.258837    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" event={"ID":"d2fc63ac-afab-4892-a059-aa5b96a80a71","Type":"ContainerDied","Data":"d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766"}
Sep 08 14:09:13.674631 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:13.674598    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:14.842555 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:14.842512    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Sep 08 14:09:14.846875 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:14.846850    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 08 14:09:16.689675 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.689652    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:09:16.802821 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.802729    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-api-access-vb25b\") pod \"d2fc63ac-afab-4892-a059-aa5b96a80a71\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") "
Sep 08 14:09:16.802999 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.802837    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-cabundle-cert\") pod \"d2fc63ac-afab-4892-a059-aa5b96a80a71\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") "
Sep 08 14:09:16.802999 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.802879    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2fc63ac-afab-4892-a059-aa5b96a80a71-proxy-tls\") pod \"d2fc63ac-afab-4892-a059-aa5b96a80a71\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") "
Sep 08 14:09:16.802999 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.802917    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2fc63ac-afab-4892-a059-aa5b96a80a71-kserve-provision-location\") pod \"d2fc63ac-afab-4892-a059-aa5b96a80a71\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") "
Sep 08 14:09:16.802999 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.802961    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-rbac-proxy-sar-config\") pod \"d2fc63ac-afab-4892-a059-aa5b96a80a71\" (UID: \"d2fc63ac-afab-4892-a059-aa5b96a80a71\") "
Sep 08 14:09:16.803309 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.803271    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2fc63ac-afab-4892-a059-aa5b96a80a71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2fc63ac-afab-4892-a059-aa5b96a80a71" (UID: "d2fc63ac-afab-4892-a059-aa5b96a80a71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:09:16.803386 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.803328    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2fc63ac-afab-4892-a059-aa5b96a80a71" (UID: "d2fc63ac-afab-4892-a059-aa5b96a80a71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:09:16.803386 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.803347    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2fc63ac-afab-4892-a059-aa5b96a80a71" (UID: "d2fc63ac-afab-4892-a059-aa5b96a80a71"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:09:16.804931 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.804911    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-api-access-vb25b" (OuterVolumeSpecName: "kube-api-access-vb25b") pod "d2fc63ac-afab-4892-a059-aa5b96a80a71" (UID: "d2fc63ac-afab-4892-a059-aa5b96a80a71"). InnerVolumeSpecName "kube-api-access-vb25b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:09:16.805039 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.805000    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2fc63ac-afab-4892-a059-aa5b96a80a71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2fc63ac-afab-4892-a059-aa5b96a80a71" (UID: "d2fc63ac-afab-4892-a059-aa5b96a80a71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:09:16.904286 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.904236    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:09:16.904286 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.904279    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2fc63ac-afab-4892-a059-aa5b96a80a71-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:09:16.904286 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.904288    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2fc63ac-afab-4892-a059-aa5b96a80a71-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:09:16.904286 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.904296    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:09:16.904286 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:16.904307    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb25b\" (UniqueName: \"kubernetes.io/projected/d2fc63ac-afab-4892-a059-aa5b96a80a71-kube-api-access-vb25b\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:09:17.277319 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.277276    2583 generic.go:356] "Generic (PLEG): container finished" podID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerID="f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b" exitCode=0
Sep 08 14:09:17.277509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.277329    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" event={"ID":"d2fc63ac-afab-4892-a059-aa5b96a80a71","Type":"ContainerDied","Data":"f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b"}
Sep 08 14:09:17.277509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.277337    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"
Sep 08 14:09:17.277509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.277379    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79" event={"ID":"d2fc63ac-afab-4892-a059-aa5b96a80a71","Type":"ContainerDied","Data":"a38305f9274ea8b4f6f9485ff6ce670f00f556603646239887a6165ed66e8ddf"}
Sep 08 14:09:17.277509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.277405    2583 scope.go:117] "RemoveContainer" containerID="d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766"
Sep 08 14:09:17.286514 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.286495    2583 scope.go:117] "RemoveContainer" containerID="f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b"
Sep 08 14:09:17.300521 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.300492    2583 scope.go:117] "RemoveContainer" containerID="4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4"
Sep 08 14:09:17.302049 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.302028    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"]
Sep 08 14:09:17.306999 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.306973    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-p8c79"]
Sep 08 14:09:17.308920 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.308902    2583 scope.go:117] "RemoveContainer" containerID="d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766"
Sep 08 14:09:17.309210 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:09:17.309191    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766\": container with ID starting with d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766 not found: ID does not exist" containerID="d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766"
Sep 08 14:09:17.309263 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.309218    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766"} err="failed to get container status \"d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766\": rpc error: code = NotFound desc = could not find container \"d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766\": container with ID starting with d00752331e0b73714d537e615cd756cc3207abd0a73e42c2e21628c10ded9766 not found: ID does not exist"
Sep 08 14:09:17.309263 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.309235    2583 scope.go:117] "RemoveContainer" containerID="f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b"
Sep 08 14:09:17.309504 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:09:17.309487    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b\": container with ID starting with f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b not found: ID does not exist" containerID="f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b"
Sep 08 14:09:17.309546 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.309511    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b"} err="failed to get container status \"f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b\": rpc error: code = NotFound desc = could not find container \"f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b\": container with ID starting with f2843cf9427ba467779ecb9c50cb9f55ea0be9ff85eb657a363a09e1d559f76b not found: ID does not exist"
Sep 08 14:09:17.309546 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.309526    2583 scope.go:117] "RemoveContainer" containerID="4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4"
Sep 08 14:09:17.309755 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:09:17.309739    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4\": container with ID starting with 4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4 not found: ID does not exist" containerID="4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4"
Sep 08 14:09:17.309795 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.309761    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4"} err="failed to get container status \"4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4\": rpc error: code = NotFound desc = could not find container \"4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4\": container with ID starting with 4ee98486d1e0cb71998ba5e6deb9a9d53f4c09a33b5c7d5f406d5a2116e45fd4 not found: ID does not exist"
Sep 08 14:09:17.595379 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:17.595349    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" path="/var/lib/kubelet/pods/d2fc63ac-afab-4892-a059-aa5b96a80a71/volumes"
Sep 08 14:09:18.137787 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.137745    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:18.138418 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.138399    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"]
Sep 08 14:09:18.138733 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.138711    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="storage-initializer"
Sep 08 14:09:18.138733 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.138730    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="storage-initializer"
Sep 08 14:09:18.138927 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.138741    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kube-rbac-proxy"
Sep 08 14:09:18.138927 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.138750    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kube-rbac-proxy"
Sep 08 14:09:18.138927 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.138763    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container"
Sep 08 14:09:18.138927 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.138770    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container"
Sep 08 14:09:18.138927 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.138879    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kube-rbac-proxy"
Sep 08 14:09:18.138927 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.138894    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2fc63ac-afab-4892-a059-aa5b96a80a71" containerName="kserve-container"
Sep 08 14:09:18.143900 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.143880    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.146709 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.146675    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 08 14:09:18.146860 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.146679    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 08 14:09:18.146998 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.146679    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 08 14:09:18.146998 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.146679    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 08 14:09:18.146998 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.146692    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-g7j4q\""
Sep 08 14:09:18.146998 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.146775    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 08 14:09:18.147233 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.147120    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 08 14:09:18.152865 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.152843    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"]
Sep 08 14:09:18.317000 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.316956    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.317210 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.317067    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.317210 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.317089    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1519a61c-e6db-415e-a0ae-b1991859c578-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.317210 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.317116    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1519a61c-e6db-415e-a0ae-b1991859c578-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.317326 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.317202    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbs2\" (UniqueName: \"kubernetes.io/projected/1519a61c-e6db-415e-a0ae-b1991859c578-kube-api-access-2cbs2\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.418323 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.418239    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.418323 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.418298    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.418514 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.418328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1519a61c-e6db-415e-a0ae-b1991859c578-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.418514 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.418354    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1519a61c-e6db-415e-a0ae-b1991859c578-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.418514 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.418414    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbs2\" (UniqueName: \"kubernetes.io/projected/1519a61c-e6db-415e-a0ae-b1991859c578-kube-api-access-2cbs2\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.418514 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:09:18.418477    2583 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 08 14:09:18.418746 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:09:18.418541    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1519a61c-e6db-415e-a0ae-b1991859c578-proxy-tls podName:1519a61c-e6db-415e-a0ae-b1991859c578 nodeName:}" failed. No retries permitted until 2026-09-08 14:09:18.918523748 +0000 UTC m=+3187.847427468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1519a61c-e6db-415e-a0ae-b1991859c578-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" (UID: "1519a61c-e6db-415e-a0ae-b1991859c578") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 08 14:09:18.418828 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.418792    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1519a61c-e6db-415e-a0ae-b1991859c578-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.419094 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.419069    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.419184 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.419134    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.429542 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.429515    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbs2\" (UniqueName: \"kubernetes.io/projected/1519a61c-e6db-415e-a0ae-b1991859c578-kube-api-access-2cbs2\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.546582 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.546534    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:09:18.923278 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.923249    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1519a61c-e6db-415e-a0ae-b1991859c578-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:18.925711 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:18.925681    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1519a61c-e6db-415e-a0ae-b1991859c578-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-d625f\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:19.057674 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:19.057637    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:19.332697 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:19.332518    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"]
Sep 08 14:09:19.335567 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:09:19.335541    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1519a61c_e6db_415e_a0ae_b1991859c578.slice/crio-2b6a922732e553c4155bbec665926c4714fecd3b6e34f27f90059f835925fbab WatchSource:0}: Error finding container 2b6a922732e553c4155bbec665926c4714fecd3b6e34f27f90059f835925fbab: Status 404 returned error can't find the container with id 2b6a922732e553c4155bbec665926c4714fecd3b6e34f27f90059f835925fbab
Sep 08 14:09:20.293620 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:20.293586    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" event={"ID":"1519a61c-e6db-415e-a0ae-b1991859c578","Type":"ContainerStarted","Data":"ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748"}
Sep 08 14:09:20.293809 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:20.293645    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" event={"ID":"1519a61c-e6db-415e-a0ae-b1991859c578","Type":"ContainerStarted","Data":"2b6a922732e553c4155bbec665926c4714fecd3b6e34f27f90059f835925fbab"}
Sep 08 14:09:24.312651 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:24.312614    2583 generic.go:356] "Generic (PLEG): container finished" podID="1519a61c-e6db-415e-a0ae-b1991859c578" containerID="ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748" exitCode=0
Sep 08 14:09:24.313050 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:24.312684    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" event={"ID":"1519a61c-e6db-415e-a0ae-b1991859c578","Type":"ContainerDied","Data":"ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748"}
Sep 08 14:09:25.319059 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:25.319003    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" event={"ID":"1519a61c-e6db-415e-a0ae-b1991859c578","Type":"ContainerStarted","Data":"6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d"}
Sep 08 14:09:25.319479 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:25.319090    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" event={"ID":"1519a61c-e6db-415e-a0ae-b1991859c578","Type":"ContainerStarted","Data":"7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e"}
Sep 08 14:09:25.319479 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:25.319356    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:25.319600 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:25.319488    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:25.320871 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:25.320842    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:09:25.338966 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:25.338915    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podStartSLOduration=3.303821306 podStartE2EDuration="7.338892807s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.035071501s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:09:25.337667759 +0000 UTC m=+3194.266571514" watchObservedRunningTime="2026-09-08 14:09:25.338892807 +0000 UTC m=+3194.267796549"
Sep 08 14:09:26.323359 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:26.323311    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:09:31.328427 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:31.328394    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:09:31.329054 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:31.329029    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:09:41.329504 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:41.329458    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:09:51.329311 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:09:51.329268    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:10:01.329070 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:01.329027    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:10:11.329664 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:11.329573    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:10:21.329316 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:21.329273    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:10:25.676500 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:25.676461    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:25.884259 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:25.884222    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:31.329082 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:31.329032    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:10:39.590778 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:39.590732    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:10:41.758336 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:41.758297    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:41.828308 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:41.828273    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:49.594653 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:49.594621    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:10:59.191544 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:59.191506    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"]
Sep 08 14:10:59.192095 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:59.191770    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" containerID="cri-o://7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e" gracePeriod=30
Sep 08 14:10:59.192095 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:59.191829    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kube-rbac-proxy" containerID="cri-o://6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d" gracePeriod=30
Sep 08 14:10:59.223377 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:59.223342    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:10:59.590940 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:59.590897    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.61:8080: connect: connection refused"
Sep 08 14:10:59.734338 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:59.734300    2583 generic.go:356] "Generic (PLEG): container finished" podID="1519a61c-e6db-415e-a0ae-b1991859c578" containerID="6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d" exitCode=2
Sep 08 14:10:59.734509 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:10:59.734350    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" event={"ID":"1519a61c-e6db-415e-a0ae-b1991859c578","Type":"ContainerDied","Data":"6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d"}
Sep 08 14:11:01.324157 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:01.324115    2583 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.61:8643/healthz\": dial tcp 10.132.0.61:8643: connect: connection refused"
Sep 08 14:11:02.942983 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.942951    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:11:02.968152 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.968046    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-cabundle-cert\") pod \"1519a61c-e6db-415e-a0ae-b1991859c578\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") "
Sep 08 14:11:02.968323 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.968169    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-kube-rbac-proxy-sar-config\") pod \"1519a61c-e6db-415e-a0ae-b1991859c578\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") "
Sep 08 14:11:02.968323 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.968227    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1519a61c-e6db-415e-a0ae-b1991859c578-kserve-provision-location\") pod \"1519a61c-e6db-415e-a0ae-b1991859c578\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") "
Sep 08 14:11:02.968323 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.968256    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cbs2\" (UniqueName: \"kubernetes.io/projected/1519a61c-e6db-415e-a0ae-b1991859c578-kube-api-access-2cbs2\") pod \"1519a61c-e6db-415e-a0ae-b1991859c578\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") "
Sep 08 14:11:02.968323 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.968297    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1519a61c-e6db-415e-a0ae-b1991859c578-proxy-tls\") pod \"1519a61c-e6db-415e-a0ae-b1991859c578\" (UID: \"1519a61c-e6db-415e-a0ae-b1991859c578\") "
Sep 08 14:11:02.968640 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.968451    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1519a61c-e6db-415e-a0ae-b1991859c578" (UID: "1519a61c-e6db-415e-a0ae-b1991859c578"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:11:02.968640 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.968627    2583 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-cabundle-cert\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:11:02.969036 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.968982    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1519a61c-e6db-415e-a0ae-b1991859c578-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1519a61c-e6db-415e-a0ae-b1991859c578" (UID: "1519a61c-e6db-415e-a0ae-b1991859c578"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:11:02.969405 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.969328    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1519a61c-e6db-415e-a0ae-b1991859c578" (UID: "1519a61c-e6db-415e-a0ae-b1991859c578"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 14:11:02.971131 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.971097    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1519a61c-e6db-415e-a0ae-b1991859c578-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1519a61c-e6db-415e-a0ae-b1991859c578" (UID: "1519a61c-e6db-415e-a0ae-b1991859c578"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 14:11:02.971358 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:02.971335    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1519a61c-e6db-415e-a0ae-b1991859c578-kube-api-access-2cbs2" (OuterVolumeSpecName: "kube-api-access-2cbs2") pod "1519a61c-e6db-415e-a0ae-b1991859c578" (UID: "1519a61c-e6db-415e-a0ae-b1991859c578"). InnerVolumeSpecName "kube-api-access-2cbs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:11:03.069281 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.069241    2583 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1519a61c-e6db-415e-a0ae-b1991859c578-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:11:03.069281 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.069270    2583 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1519a61c-e6db-415e-a0ae-b1991859c578-kserve-provision-location\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:11:03.069281 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.069281    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbs2\" (UniqueName: \"kubernetes.io/projected/1519a61c-e6db-415e-a0ae-b1991859c578-kube-api-access-2cbs2\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:11:03.069281 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.069291    2583 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1519a61c-e6db-415e-a0ae-b1991859c578-proxy-tls\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:11:03.758003 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.757970    2583 generic.go:356] "Generic (PLEG): container finished" podID="1519a61c-e6db-415e-a0ae-b1991859c578" containerID="7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e" exitCode=0
Sep 08 14:11:03.758199 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.758052    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"
Sep 08 14:11:03.758199 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.758054    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" event={"ID":"1519a61c-e6db-415e-a0ae-b1991859c578","Type":"ContainerDied","Data":"7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e"}
Sep 08 14:11:03.758199 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.758184    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f" event={"ID":"1519a61c-e6db-415e-a0ae-b1991859c578","Type":"ContainerDied","Data":"2b6a922732e553c4155bbec665926c4714fecd3b6e34f27f90059f835925fbab"}
Sep 08 14:11:03.758307 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.758213    2583 scope.go:117] "RemoveContainer" containerID="6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d"
Sep 08 14:11:03.766773 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.766750    2583 scope.go:117] "RemoveContainer" containerID="7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e"
Sep 08 14:11:03.774564 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.774542    2583 scope.go:117] "RemoveContainer" containerID="ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748"
Sep 08 14:11:03.778150 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.778123    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"]
Sep 08 14:11:03.782837 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.782807    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-d625f"]
Sep 08 14:11:03.783803 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.783782    2583 scope.go:117] "RemoveContainer" containerID="6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d"
Sep 08 14:11:03.784190 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:11:03.784171    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d\": container with ID starting with 6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d not found: ID does not exist" containerID="6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d"
Sep 08 14:11:03.784240 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.784201    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d"} err="failed to get container status \"6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d\": rpc error: code = NotFound desc = could not find container \"6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d\": container with ID starting with 6565a5e0d9f3abf03f7cf417defb961269e560540602d9f8024d49fa54dfc77d not found: ID does not exist"
Sep 08 14:11:03.784240 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.784222    2583 scope.go:117] "RemoveContainer" containerID="7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e"
Sep 08 14:11:03.784500 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:11:03.784481    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e\": container with ID starting with 7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e not found: ID does not exist" containerID="7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e"
Sep 08 14:11:03.784550 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.784505    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e"} err="failed to get container status \"7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e\": rpc error: code = NotFound desc = could not find container \"7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e\": container with ID starting with 7417b5d4b91fb2a68a522e01aa88cfb056e134d9cbed62d6699fb436a0c3e57e not found: ID does not exist"
Sep 08 14:11:03.784550 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.784522    2583 scope.go:117] "RemoveContainer" containerID="ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748"
Sep 08 14:11:03.784763 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:11:03.784750    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748\": container with ID starting with ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748 not found: ID does not exist" containerID="ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748"
Sep 08 14:11:03.784813 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:03.784764    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748"} err="failed to get container status \"ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748\": rpc error: code = NotFound desc = could not find container \"ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748\": container with ID starting with ce74f325c114170c5e92a8ca350cade1e2382165f2ee48fe4ea9a6e3dd194748 not found: ID does not exist"
Sep 08 14:11:05.595744 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.595712    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" path="/var/lib/kubelet/pods/1519a61c-e6db-415e-a0ae-b1991859c578/volumes"
Sep 08 14:11:05.812840 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.812807    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vgfwk/must-gather-57rbd"]
Sep 08 14:11:05.813163 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.813151    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="storage-initializer"
Sep 08 14:11:05.813224 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.813165    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="storage-initializer"
Sep 08 14:11:05.813224 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.813175    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kube-rbac-proxy"
Sep 08 14:11:05.813224 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.813182    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kube-rbac-proxy"
Sep 08 14:11:05.813224 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.813193    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container"
Sep 08 14:11:05.813224 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.813198    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container"
Sep 08 14:11:05.813384 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.813262    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kube-rbac-proxy"
Sep 08 14:11:05.813384 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.813270    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="1519a61c-e6db-415e-a0ae-b1991859c578" containerName="kserve-container"
Sep 08 14:11:05.817979 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.817952    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:05.819376 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.819349    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:05.820365 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.820347    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vgfwk\"/\"openshift-service-ca.crt\""
Sep 08 14:11:05.820502 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.820362    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vgfwk\"/\"kube-root-ca.crt\""
Sep 08 14:11:05.820502 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.820380    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vgfwk\"/\"default-dockercfg-n7vwl\""
Sep 08 14:11:05.826620 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.826595    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vgfwk/must-gather-57rbd"]
Sep 08 14:11:05.893446 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.893349    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-must-gather-output\") pod \"must-gather-57rbd\" (UID: \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\") " pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:05.893446 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.893406    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb592\" (UniqueName: \"kubernetes.io/projected/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-kube-api-access-sb592\") pod \"must-gather-57rbd\" (UID: \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\") " pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:05.920028 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.919980    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:05.994908 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.994869    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-must-gather-output\") pod \"must-gather-57rbd\" (UID: \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\") " pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:05.995120 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.994920    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb592\" (UniqueName: \"kubernetes.io/projected/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-kube-api-access-sb592\") pod \"must-gather-57rbd\" (UID: \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\") " pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:05.995250 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:05.995232    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-must-gather-output\") pod \"must-gather-57rbd\" (UID: \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\") " pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:06.003996 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:06.003969    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb592\" (UniqueName: \"kubernetes.io/projected/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-kube-api-access-sb592\") pod \"must-gather-57rbd\" (UID: \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\") " pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:06.140804 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:06.140776    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:06.416372 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:06.416286    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vgfwk/must-gather-57rbd"]
Sep 08 14:11:06.419914 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:11:06.419882    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b784503_bfd0_41d5_83c5_cfc25bb5c0b6.slice/crio-a6c159abef91c47c46fe09065b5411e68c36efefe05fa44eef90eda02776b4af WatchSource:0}: Error finding container a6c159abef91c47c46fe09065b5411e68c36efefe05fa44eef90eda02776b4af: Status 404 returned error can't find the container with id a6c159abef91c47c46fe09065b5411e68c36efefe05fa44eef90eda02776b4af
Sep 08 14:11:06.421859 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:06.421841    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 14:11:06.777380 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:06.777286    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vgfwk/must-gather-57rbd" event={"ID":"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6","Type":"ContainerStarted","Data":"a6c159abef91c47c46fe09065b5411e68c36efefe05fa44eef90eda02776b4af"}
Sep 08 14:11:11.817110 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:11.817078    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 14:11:11.823665 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:11.823594    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 14:11:17.835171 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:17.835129    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vgfwk/must-gather-57rbd" event={"ID":"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6","Type":"ContainerStarted","Data":"e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073"}
Sep 08 14:11:17.835569 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:17.835186    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vgfwk/must-gather-57rbd" event={"ID":"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6","Type":"ContainerStarted","Data":"38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d"}
Sep 08 14:11:17.857145 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:17.857096    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vgfwk/must-gather-57rbd" podStartSLOduration=2.446472484 podStartE2EDuration="12.857079057s" totalImagesPullingTime="10.410606573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:11:17.855650826 +0000 UTC m=+3306.784554569" watchObservedRunningTime="2026-09-08 14:11:17.857079057 +0000 UTC m=+3306.785982800"
Sep 08 14:11:29.103258 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:29.103217    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:29.203930 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:29.203877    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:45.976736 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:45.976699    2583 generic.go:356] "Generic (PLEG): container finished" podID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" containerID="38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d" exitCode=0
Sep 08 14:11:45.977184 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:45.976774    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vgfwk/must-gather-57rbd" event={"ID":"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6","Type":"ContainerDied","Data":"38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d"}
Sep 08 14:11:45.977184 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:45.977100    2583 scope.go:117] "RemoveContainer" containerID="38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d"
Sep 08 14:11:46.837337 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:46.837297    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vgfwk_must-gather-57rbd_0b784503-bfd0-41d5-83c5-cfc25bb5c0b6/gather/0.log"
Sep 08 14:11:47.427459 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:47.427414    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:47.575911 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:47.575858    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:47.672398 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:47.672361    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:50.525644 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:50.525614    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jr9bb_de8493b4-7db2-438d-bcaa-b97e00b8c2e3/global-pull-secret-syncer/0.log"
Sep 08 14:11:50.645087 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:50.645052    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log"
Sep 08 14:11:50.813789 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:50.813760    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-66.ec2.internal_727e83fa5ccafe51a7ba3765918be5a3/haproxy/0.log"
Sep 08 14:11:52.396271 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.396233    2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-vgfwk/must-gather-57rbd"]
Sep 08 14:11:52.396678 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.396422    2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-vgfwk/must-gather-57rbd" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" containerName="copy" containerID="cri-o://e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073" gracePeriod=2
Sep 08 14:11:52.402507 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:11:52.402473    2583 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"openshift-must-gather-vgfwk\"/\"must-gather-57rbd\": pods \"must-gather-57rbd\" not found" pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:52.402937 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.402915    2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-vgfwk/must-gather-57rbd"]
Sep 08 14:11:52.629665 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.629640    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vgfwk_must-gather-57rbd_0b784503-bfd0-41d5-83c5-cfc25bb5c0b6/copy/0.log"
Sep 08 14:11:52.631170 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.631153    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:52.632804 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:11:52.632781    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-57rbd\" is forbidden: User \"system:node:ip-10-0-137-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vgfwk\": no relationship found between node 'ip-10-0-137-66.ec2.internal' and this object" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:52.701130 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.701042    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-must-gather-output\") pod \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\" (UID: \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\") "
Sep 08 14:11:52.701130 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.701129    2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sb592\" (UniqueName: \"kubernetes.io/projected/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-kube-api-access-sb592\") pod \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\" (UID: \"0b784503-bfd0-41d5-83c5-cfc25bb5c0b6\") "
Sep 08 14:11:52.702424 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.702395    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" (UID: "0b784503-bfd0-41d5-83c5-cfc25bb5c0b6"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 14:11:52.703533 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.703507    2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-kube-api-access-sb592" (OuterVolumeSpecName: "kube-api-access-sb592") pod "0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" (UID: "0b784503-bfd0-41d5-83c5-cfc25bb5c0b6"). InnerVolumeSpecName "kube-api-access-sb592". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 14:11:52.802113 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.802077    2583 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-must-gather-output\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:11:52.802113 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:52.802110    2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb592\" (UniqueName: \"kubernetes.io/projected/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6-kube-api-access-sb592\") on node \"ip-10-0-137-66.ec2.internal\" DevicePath \"\""
Sep 08 14:11:53.011169 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.011089    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-vgfwk_must-gather-57rbd_0b784503-bfd0-41d5-83c5-cfc25bb5c0b6/copy/0.log"
Sep 08 14:11:53.011517 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.011493    2583 generic.go:356] "Generic (PLEG): container finished" podID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" containerID="e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073" exitCode=143
Sep 08 14:11:53.011586 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.011537    2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:53.011586 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.011547    2583 scope.go:117] "RemoveContainer" containerID="e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073"
Sep 08 14:11:53.013492 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:11:53.013462    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-57rbd\" is forbidden: User \"system:node:ip-10-0-137-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vgfwk\": no relationship found between node 'ip-10-0-137-66.ec2.internal' and this object" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:53.020420 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.020403    2583 scope.go:117] "RemoveContainer" containerID="38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d"
Sep 08 14:11:53.022023 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:11:53.021984    2583 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-57rbd\" is forbidden: User \"system:node:ip-10-0-137-66.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-vgfwk\": no relationship found between node 'ip-10-0-137-66.ec2.internal' and this object" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" pod="openshift-must-gather-vgfwk/must-gather-57rbd"
Sep 08 14:11:53.032250 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.032229    2583 scope.go:117] "RemoveContainer" containerID="e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073"
Sep 08 14:11:53.032567 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:11:53.032550    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073\": container with ID starting with e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073 not found: ID does not exist" containerID="e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073"
Sep 08 14:11:53.032618 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.032577    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073"} err="failed to get container status \"e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073\": rpc error: code = NotFound desc = could not find container \"e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073\": container with ID starting with e035201ea449e2596963f446fc8152390d34760e3a721d02ac11625a73ca9073 not found: ID does not exist"
Sep 08 14:11:53.032618 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.032594    2583 scope.go:117] "RemoveContainer" containerID="38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d"
Sep 08 14:11:53.032871 ip-10-0-137-66 kubenswrapper[2583]: E0908 14:11:53.032852    2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d\": container with ID starting with 38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d not found: ID does not exist" containerID="38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d"
Sep 08 14:11:53.032927 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.032876    2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d"} err="failed to get container status \"38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d\": rpc error: code = NotFound desc = could not find container \"38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d\": container with ID starting with 38802edaf30619851636f36c283cd36306d1d1f1a1392e74b143960626ad019d not found: ID does not exist"
Sep 08 14:11:53.320347 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.320300    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:53.398593 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.398553    2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-78brt_92745d4d-0c99-4d1b-a91e-b7a546e0e583/konnectivity-agent/0.log" limit=1024
Sep 08 14:11:53.594884 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:53.594790    2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" path="/var/lib/kubelet/pods/0b784503-bfd0-41d5-83c5-cfc25bb5c0b6/volumes"
Sep 08 14:11:54.778044 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:54.778003    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b45e756-c32f-46e3-8cee-11ef1c8e583b/alertmanager/0.log"
Sep 08 14:11:54.801522 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:54.801495    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b45e756-c32f-46e3-8cee-11ef1c8e583b/config-reloader/0.log"
Sep 08 14:11:54.827820 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:54.827796    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b45e756-c32f-46e3-8cee-11ef1c8e583b/kube-rbac-proxy-web/0.log"
Sep 08 14:11:54.851566 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:54.851540    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b45e756-c32f-46e3-8cee-11ef1c8e583b/kube-rbac-proxy/0.log"
Sep 08 14:11:54.877843 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:54.877813    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b45e756-c32f-46e3-8cee-11ef1c8e583b/kube-rbac-proxy-metric/0.log"
Sep 08 14:11:54.897858 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:54.897827    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b45e756-c32f-46e3-8cee-11ef1c8e583b/prom-label-proxy/0.log"
Sep 08 14:11:54.919125 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:54.919095    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_9b45e756-c32f-46e3-8cee-11ef1c8e583b/init-config-reloader/0.log"
Sep 08 14:11:54.966757 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:54.966683    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-s657h_ba4b3c60-a06b-4af3-9d0a-e80226cae5ab/cluster-monitoring-operator/0.log"
Sep 08 14:11:55.002414 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.002390    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mdxjp_90b4ca0f-0a3d-449c-86cf-0df165ac2ee8/kube-state-metrics/0.log"
Sep 08 14:11:55.024685 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.024654    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mdxjp_90b4ca0f-0a3d-449c-86cf-0df165ac2ee8/kube-rbac-proxy-main/0.log"
Sep 08 14:11:55.050815 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.050793    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-mdxjp_90b4ca0f-0a3d-449c-86cf-0df165ac2ee8/kube-rbac-proxy-self/0.log"
Sep 08 14:11:55.156574 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.156545    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bnm9h_3ce5a7b7-4478-451d-b4fd-fc88df9f91de/node-exporter/0.log"
Sep 08 14:11:55.176637 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.176613    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bnm9h_3ce5a7b7-4478-451d-b4fd-fc88df9f91de/kube-rbac-proxy/0.log"
Sep 08 14:11:55.198975 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.198927    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bnm9h_3ce5a7b7-4478-451d-b4fd-fc88df9f91de/init-textfile/0.log"
Sep 08 14:11:55.685871 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.685842    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-2pf9c_65090ac4-f95f-42ba-8c98-a8455ab7e7c3/prometheus-operator/0.log"
Sep 08 14:11:55.708966 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.708933    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-2pf9c_65090ac4-f95f-42ba-8c98-a8455ab7e7c3/kube-rbac-proxy/0.log"
Sep 08 14:11:55.769727 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.769696    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bf894ccb-sx6wb_63c6e89b-db17-4ab1-b8bf-815a19936cff/telemeter-client/0.log"
Sep 08 14:11:55.793173 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.793146    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bf894ccb-sx6wb_63c6e89b-db17-4ab1-b8bf-815a19936cff/reload/0.log"
Sep 08 14:11:55.814809 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:55.814780    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79bf894ccb-sx6wb_63c6e89b-db17-4ab1-b8bf-815a19936cff/kube-rbac-proxy/0.log"
Sep 08 14:11:57.894442 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:57.894416    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/2.log"
Sep 08 14:11:57.902380 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:57.902348    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-8rdfw_574adb2c-aa3b-486f-bf43-06720a5f134f/console-operator/3.log"
Sep 08 14:11:58.373807 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:58.373775    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-nxwsj_ec637e70-0655-4e0f-9da2-a4025cfd7e0f/download-server/0.log"
Sep 08 14:11:59.795308 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.795273    2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"]
Sep 08 14:11:59.795700 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.795618    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" containerName="copy"
Sep 08 14:11:59.795700 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.795629    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" containerName="copy"
Sep 08 14:11:59.795700 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.795637    2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" containerName="gather"
Sep 08 14:11:59.795700 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.795643    2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" containerName="gather"
Sep 08 14:11:59.795700 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.795700    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" containerName="copy"
Sep 08 14:11:59.795887 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.795709    2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b784503-bfd0-41d5-83c5-cfc25bb5c0b6" containerName="gather"
Sep 08 14:11:59.797832 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.797815    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.799807 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.799785    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcsfc\"/\"kube-root-ca.crt\""
Sep 08 14:11:59.800276 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.800258    2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcsfc\"/\"openshift-service-ca.crt\""
Sep 08 14:11:59.800382 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.800276    2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tcsfc\"/\"default-dockercfg-dms7m\""
Sep 08 14:11:59.806570 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.806543    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"]
Sep 08 14:11:59.861189 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.861149    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-sys\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.861189 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.861192    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qddwn\" (UniqueName: \"kubernetes.io/projected/795e0a87-0e6c-4796-92be-b1326e5e7a9c-kube-api-access-qddwn\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.861411 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.861253    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-proc\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.861411 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.861316    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-lib-modules\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.861411 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.861341    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-podres\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.961983 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.961947    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-proc\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.962188 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.962034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-lib-modules\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.962188 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.962064    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-podres\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.962188 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.962090    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-sys\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.962188 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.962103    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-proc\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.962188 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.962110    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qddwn\" (UniqueName: \"kubernetes.io/projected/795e0a87-0e6c-4796-92be-b1326e5e7a9c-kube-api-access-qddwn\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.962380 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.962202    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-sys\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.962380 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.962208    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-lib-modules\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.962380 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.962211    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/795e0a87-0e6c-4796-92be-b1326e5e7a9c-podres\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.970197 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.970166    2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qddwn\" (UniqueName: \"kubernetes.io/projected/795e0a87-0e6c-4796-92be-b1326e5e7a9c-kube-api-access-qddwn\") pod \"perf-node-gather-daemonset-z6pq5\" (UID: \"795e0a87-0e6c-4796-92be-b1326e5e7a9c\") " pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:11:59.999296 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:11:59.999253    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k6zh9_8e6529bc-e29e-406f-ae0e-55c85d81465b/dns/0.log"
Sep 08 14:12:00.018733 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:00.018707    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k6zh9_8e6529bc-e29e-406f-ae0e-55c85d81465b/kube-rbac-proxy/0.log"
Sep 08 14:12:00.069846 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:00.069815    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l9frn_02703e20-c856-4652-9e02-6086e7b11179/dns-node-resolver/0.log"
Sep 08 14:12:00.111556 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:00.111528    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:12:00.383553 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:00.383526    2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"]
Sep 08 14:12:00.386032 ip-10-0-137-66 kubenswrapper[2583]: W0908 14:12:00.385984    2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod795e0a87_0e6c_4796_92be_b1326e5e7a9c.slice/crio-a16a04befcd19b817d520ec04e978b7f8801ecd8ca5232e23a0e1c8acd765303 WatchSource:0}: Error finding container a16a04befcd19b817d520ec04e978b7f8801ecd8ca5232e23a0e1c8acd765303: Status 404 returned error can't find the container with id a16a04befcd19b817d520ec04e978b7f8801ecd8ca5232e23a0e1c8acd765303
Sep 08 14:12:00.681840 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:00.681753    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vrfh7_817efad1-a9dc-4e98-ae2d-396f97d0eda9/node-ca/0.log"
Sep 08 14:12:01.054283 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:01.054201    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5" event={"ID":"795e0a87-0e6c-4796-92be-b1326e5e7a9c","Type":"ContainerStarted","Data":"0d309e59de5e7ab88219e3a176bb1b07a1b308868a8a308388b7788d6e22dcef"}
Sep 08 14:12:01.054283 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:01.054274    2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5" event={"ID":"795e0a87-0e6c-4796-92be-b1326e5e7a9c","Type":"ContainerStarted","Data":"a16a04befcd19b817d520ec04e978b7f8801ecd8ca5232e23a0e1c8acd765303"}
Sep 08 14:12:01.054676 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:01.054344    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:12:01.073273 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:01.073205    2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5" podStartSLOduration=2.073183669 podStartE2EDuration="2.073183669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:12:01.070183681 +0000 UTC m=+3349.999087446" watchObservedRunningTime="2026-09-08 14:12:01.073183669 +0000 UTC m=+3350.002087412"
Sep 08 14:12:01.435633 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:01.435601    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-567cf589fd-rkshv_dabbf1a4-723b-4ccc-b631-ab3db3d1ff05/router/0.log"
Sep 08 14:12:01.815686 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:01.815656    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9qg7h_a352c5b0-060c-4938-9775-9138f5461a28/serve-healthcheck-canary/0.log"
Sep 08 14:12:02.182929 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:02.182803    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-6lrnr_da748d4c-3623-4746-b899-479c9aaa81c1/insights-operator/0.log"
Sep 08 14:12:02.183985 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:02.183962    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-6lrnr_da748d4c-3623-4746-b899-479c9aaa81c1/insights-operator/1.log"
Sep 08 14:12:02.352923 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:02.352892    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rpgqv_12dbfc49-1b02-4e95-9c56-91a93b3054f4/kube-rbac-proxy/0.log"
Sep 08 14:12:02.373334 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:02.373299    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rpgqv_12dbfc49-1b02-4e95-9c56-91a93b3054f4/exporter/0.log"
Sep 08 14:12:02.394632 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:02.394607    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rpgqv_12dbfc49-1b02-4e95-9c56-91a93b3054f4/extractor/0.log"
Sep 08 14:12:04.852476 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:04.852441    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-575d6764f4-7tqb9_952814a1-83ac-44c3-9ca5-75fa210b9f36/manager/0.log"
Sep 08 14:12:05.413220 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:05.413179    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-99gt9_fc99e657-db72-42b1-a493-fd0e2c93a2fe/manager/0.log"
Sep 08 14:12:05.631277 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:05.631239    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-whg7m_f425f0b6-8c98-47fc-b79f-881748983b61/seaweedfs-tls-custom/0.log"
Sep 08 14:12:07.069763 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:07.069735    2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tcsfc/perf-node-gather-daemonset-z6pq5"
Sep 08 14:12:11.407334 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.407228    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mw5r7_a5dbb570-2303-4ba8-848c-b8ac08d60865/kube-multus-additional-cni-plugins/0.log"
Sep 08 14:12:11.429852 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.429824    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mw5r7_a5dbb570-2303-4ba8-848c-b8ac08d60865/egress-router-binary-copy/0.log"
Sep 08 14:12:11.479995 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.479964    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mw5r7_a5dbb570-2303-4ba8-848c-b8ac08d60865/cni-plugins/0.log"
Sep 08 14:12:11.501174 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.501148    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mw5r7_a5dbb570-2303-4ba8-848c-b8ac08d60865/bond-cni-plugin/0.log"
Sep 08 14:12:11.521623 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.521595    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mw5r7_a5dbb570-2303-4ba8-848c-b8ac08d60865/routeoverride-cni/0.log"
Sep 08 14:12:11.542917 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.542875    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mw5r7_a5dbb570-2303-4ba8-848c-b8ac08d60865/whereabouts-cni-bincopy/0.log"
Sep 08 14:12:11.593790 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.593383    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mw5r7_a5dbb570-2303-4ba8-848c-b8ac08d60865/whereabouts-cni/0.log"
Sep 08 14:12:11.640278 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.640245    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-szhm7_d533862a-b3ec-4f39-9018-e357ba3f02c5/kube-multus/0.log"
Sep 08 14:12:11.771312 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.771232    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-82vzh_29de17e1-0cb6-41fb-bd61-0a1fc39b19b3/network-metrics-daemon/0.log"
Sep 08 14:12:11.794631 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:11.794601    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-82vzh_29de17e1-0cb6-41fb-bd61-0a1fc39b19b3/kube-rbac-proxy/0.log"
Sep 08 14:12:12.779203 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:12.779177    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b2fbc_5d710a10-f153-4cad-9c8f-2482d1a4ebef/ovn-controller/0.log"
Sep 08 14:12:12.794060 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:12.794037    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b2fbc_5d710a10-f153-4cad-9c8f-2482d1a4ebef/ovn-acl-logging/0.log"
Sep 08 14:12:12.820834 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:12.820794    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b2fbc_5d710a10-f153-4cad-9c8f-2482d1a4ebef/kube-rbac-proxy-node/0.log"
Sep 08 14:12:12.841649 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:12.841623    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b2fbc_5d710a10-f153-4cad-9c8f-2482d1a4ebef/kube-rbac-proxy-ovn-metrics/0.log"
Sep 08 14:12:12.861389 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:12.861337    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b2fbc_5d710a10-f153-4cad-9c8f-2482d1a4ebef/northd/0.log"
Sep 08 14:12:12.880400 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:12.880373    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b2fbc_5d710a10-f153-4cad-9c8f-2482d1a4ebef/nbdb/0.log"
Sep 08 14:12:12.899830 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:12.899800    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b2fbc_5d710a10-f153-4cad-9c8f-2482d1a4ebef/sbdb/0.log"
Sep 08 14:12:12.960970 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:12.960936    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-b2fbc_5d710a10-f153-4cad-9c8f-2482d1a4ebef/ovnkube-controller/0.log"
Sep 08 14:12:14.206457 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:14.206433    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-44vpt_374c967a-24a3-40b4-872f-bfa954850357/network-check-target-container/0.log"
Sep 08 14:12:15.175896 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:15.175869    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-98nf9_b546af0c-d8b5-4ce6-a934-52eeab3ef44d/iptables-alerter/0.log"
Sep 08 14:12:15.893973 ip-10-0-137-66 kubenswrapper[2583]: I0908 14:12:15.893942    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-ch66t_a4b6680b-cc21-46ee-a842-48b689876d97/tuned/0.log"