Sep 09 20:51:50.328036 ip-10-0-130-40 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:51:50.734117 ip-10-0-130-40 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:51:50.734117 ip-10-0-130-40 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:51:50.734117 ip-10-0-130-40 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:51:50.734117 ip-10-0-130-40 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:51:50.734117 ip-10-0-130-40 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:51:50.735103 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.735022    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:51:50.740815 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740799    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:51:50.740815 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740815    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740819    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740822    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740825    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740827    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740830    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740832    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740841    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740844    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740846    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740849    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740854    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740857    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740861    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740863    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740866    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740869    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740871    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740874    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:51:50.740881 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740877    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740879    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740883    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740886    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740889    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740893    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740897    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740901    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740904    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740907    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740910    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740913    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740915    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740918    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740921    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740923    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740926    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740929    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740932    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:51:50.741376 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740934    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740936    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740939    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740942    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740944    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740947    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740949    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740951    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740954    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740956    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740959    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740961    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740964    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740968    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740971    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740973    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740976    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740979    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740981    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740984    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:51:50.741872 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740986    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740989    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740991    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.740994    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741001    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741004    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741006    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741009    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741011    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741014    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741016    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741018    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741021    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741023    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741026    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741028    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741031    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741033    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741036    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741038    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:51:50.742480 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741040    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741043    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741045    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741048    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741050    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741053    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741056    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741058    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741061    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741063    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741066    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741068    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741071    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741073    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741076    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741078    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741081    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741084    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741086    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741088    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:51:50.743327 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741792    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741800    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741803    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741805    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741809    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741812    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741815    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741819    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741823    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741826    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741829    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741832    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741834    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741837    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741839    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741841    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741844    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741846    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741850    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:51:50.744160 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741853    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741855    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741858    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741860    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741863    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741865    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741869    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741872    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741875    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741877    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741880    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741884    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741886    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741889    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741892    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741894    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741896    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741899    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741901    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741903    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:51:50.745018 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741906    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741909    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741911    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741914    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741917    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741919    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741922    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741925    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741927    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741930    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741932    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741935    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741939    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741942    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741944    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741947    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741949    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741952    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741955    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741957    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:51:50.745914 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741960    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741962    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741965    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741967    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741970    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741973    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741976    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741978    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741980    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741983    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741985    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741988    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741990    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741993    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741995    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.741997    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742000    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742002    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742004    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:51:50.746422 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742007    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742009    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742012    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742015    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742017    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742020    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742023    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742026    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742029    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742032    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742034    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742037    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742040    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742042    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742045    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742049    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742052    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742054    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742057    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742060    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:51:50.747043 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.742062    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743878    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743893    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743898    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743903    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743907    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743911    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743915    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743919    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743924    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743928    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743933    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743937    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743942    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743946    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743950    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743954    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743958    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743963    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743967    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:51:50.747538 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743971    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743975    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743980    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743984    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743988    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743992    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.743999    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744005    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744009    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744013    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744017    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744021    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744025    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744035    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744040    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744044    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744048    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744052    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744057    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:51:50.748065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744061    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744065    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744069    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744073    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744077    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744081    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744085    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744090    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744094    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744098    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744102    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744106    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744110    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744114    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744120    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744124    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744128    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744132    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744136    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:51:50.748574 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744140    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744144    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744148    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744152    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744156    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744159    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744163    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744168    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744173    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744177    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744180    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744185    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744189    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744193    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744200    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744207    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744212    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744217    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744225    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:51:50.749054 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744231    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744236    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744241    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744246    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744251    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744256    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744261    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744266    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744270    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744274    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744278    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744283    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744288    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744293    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744297    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744301    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744305    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744309    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744314    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744318    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:51:50.749501 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744323    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744327    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.744331    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744441    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744494    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744508    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744514    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744521    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744526    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744533    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744539    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744544    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744549    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744554    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744560    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744565    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744570    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744574    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744579    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744585    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744590    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744596    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744601    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744605    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:51:50.750055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744611    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744616    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744622    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744626    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744631    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744637    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744641    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744646    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744651    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744657    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744661    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744668    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744673    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744677    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744682    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744687    2580 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744692    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744698    2580 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744704    2580 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744708    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744713    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744718    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744723    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744729    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744733    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:51:50.750628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744738    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744743    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744747    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744753    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744775    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744780    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744784    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744789    2580 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744795    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744799    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744804    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744809    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744813    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744818    2580 flags.go:64] FLAG: --help="false"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744823    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-130-40.ec2.internal"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744828    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744835    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744839    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744845    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744850    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744854    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744859    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744864    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:51:50.751247 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744868    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744873    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744878    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744883    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744887    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744892    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744896    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744903    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744908    2580 flags.go:64] FLAG: --lock-file=""
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744912    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744917    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744922    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744930    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744936    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744940    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744945    2580 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744951    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744956    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744961    2580 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744965    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744972    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744976    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744982    2580 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744988    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744992    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:51:50.751850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.744998    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745003    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745007    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745012    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745016    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745028    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745033    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745038    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745042    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745047    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745055    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745060    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745065    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745070    2580 flags.go:64] FLAG: --port="10250"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745076    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745081    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eaf618320345de24"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745086    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745091    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745095    2580 flags.go:64] FLAG: --register-node="true"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745100    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745105    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745111    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745115    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745120    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745125    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:51:50.752456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745130    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745135    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745139    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745144    2580 flags.go:64] FLAG: --runonce="false"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745148    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745153    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745158    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745163    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745170    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745174    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745179    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745184    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745189    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745193    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745198    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745202    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745207    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745212    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745216    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745225    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745229    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745236    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745241    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745245    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745250    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:51:50.753087 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745254    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745258    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745263    2580 flags.go:64] FLAG: --v="2"
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745269    2580 flags.go:64] FLAG: --version="false"
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745277    2580 flags.go:64] FLAG: --vmodule=""
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745282    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.745287    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745468    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745474    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745480    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745485    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745490    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745497    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745503    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745508    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745514    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745519    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745523    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745527    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745531    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745535    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:51:50.753770 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745539    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745544    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745548    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745552    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745555    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745559    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745563    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745569    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745573    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745577    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745581    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745585    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745589    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745593    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745597    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745601    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745606    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745610    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745614    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745618    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:51:50.754293 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745622    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745626    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745631    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745635    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745639    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745643    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745649    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745653    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745657    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745661    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745665    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745669    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745673    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745677    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745681    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745685    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745689    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745695    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745700    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:51:50.754806 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745707    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745712    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745716    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745720    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745724    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745728    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745732    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745736    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745741    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745745    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745749    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745753    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745776    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745780    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745784    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745789    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745793    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745796    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745800    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745810    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:51:50.755273 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745816    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745821    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745825    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745829    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745833    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745837    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745841    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745845    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745850    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745853    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745858    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745862    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745868    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745872    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745875    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745880    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745884    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745888    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745893    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745897    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:51:50.755799 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745901    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745905    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745909    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745914    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745918    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.745921    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.746437    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.755185    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.755200    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.755219    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.755225    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755279    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755286    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755290    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:51:50.756288 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755293    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755296    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755299    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755302    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755304    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755307    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755309    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755312    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755314    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755317    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755319    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755322    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755325    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755327    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755329    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755332    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755334    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755337    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755339    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755342    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:51:50.756654 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755344    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755348    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755350    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755353    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755355    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755357    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755360    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755364    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755366    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755369    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755371    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755374    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755376    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755379    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755381    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755384    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755386    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755388    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755391    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755393    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:51:50.757196 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755396    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755398    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755401    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755403    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755406    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755408    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755411    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755413    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755415    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755418    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755420    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755423    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755425    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755427    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755430    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755434    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755438    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755441    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755445    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:51:50.757684 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755447    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755449    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755452    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755455    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755457    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755460    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755462    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755465    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755467    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755470    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755472    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755475    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755477    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755480    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755482    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755485    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755487    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755490    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755492    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:51:50.758161 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755495    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755497    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755500    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755504    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755507    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755510    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755512    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755515    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755517    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755521    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755524    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755526    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755529    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755532    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755535    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755538    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755540    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:51:50.758630 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755543    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.755547    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755657    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755662    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755665    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755668    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755671    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755673    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755676    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755679    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755682    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755685    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755688    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755690    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755692    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:51:50.759072 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755695    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755697    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755700    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755702    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755705    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755707    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755710    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755713    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755715    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755717    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755721    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755724    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755728    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755731    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755734    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755737    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755740    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755743    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755745    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755748    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:51:50.759437 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755750    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755753    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755770    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755773    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755776    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755778    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755781    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755783    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755787    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755791    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755793    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755796    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755799    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755802    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755805    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755808    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755810    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755813    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755816    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:51:50.759933 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755818    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755821    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755823    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755826    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755829    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755832    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755834    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755837    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755839    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755841    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755844    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755846    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755849    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755851    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755854    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755856    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755858    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755861    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755864    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755868    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:51:50.760402 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755870    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755873    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755875    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755878    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755880    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755882    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755885    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755887    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755889    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755892    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755894    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755897    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755899    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755901    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755904    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755906    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755909    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755912    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755914    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:51:50.760898 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755917    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755919    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755922    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755924    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755927    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755929    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755932    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:50.755934    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.755939    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.756039    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.758131    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.759154    2580 server.go:1019] "Starting client certificate rotation"
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.759249    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:51:50.761341 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.759279    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:51:50.761912 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.761897    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:51:50.763062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.763044    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:51:50.774062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.774019    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:51:50.777278 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.777260    2580 log.go:25] "Validated CRI v1 image API"
Sep 09 20:51:50.778092 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.778066    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:51:50.781011 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.780985    2580 fs.go:135] Filesystem UUIDs: map[2a731ea8-5164-448e-8689-edc091d19e1b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ae7ffedf-2a21-4528-afb6-4fe04bc3a612:/dev/nvme0n1p4]
Sep 09 20:51:50.781011 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.781005    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:51:50.785892 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.785772    2580 manager.go:216] Machine: {Timestamp:2026-09-09 20:51:50.784526077 +0000 UTC m=+0.345553385 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100633 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26351fd310edccf870976cba699e1b SystemUUID:ec26351f-d310-edcc-f870-976cba699e1b BootID:640fb6d5-5a89-427d-bf17-71a6b8cba698 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:49:c0:24:b4:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:49:c0:24:b4:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:17:35:38:e6:61 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:51:50.785892 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.785887    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:51:50.786007 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.785966    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:51:50.788798 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.788772    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:51:50.788930 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.788801    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-40.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:51:50.789414 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.789405    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:51:50.789447 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.789417    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:51:50.789447 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.789425    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:51:50.790101 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.790090    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:51:50.791230 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.791220    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:51:50.792703 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.792686    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:51:50.794722 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.794708    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:51:50.794854 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.794844    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:51:50.794894 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.794875    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:51:50.794894 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.794885    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:51:50.794945 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.794898    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:51:50.794945 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.794907    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:51:50.794945 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.794916    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:51:50.795259 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.795247    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:51:50.795324 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.795266    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:51:50.797257 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.797241    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:51:50.799412 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.799398    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:51:50.799461 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.799430    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:51:50.800174 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800163    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:51:50.800212 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800181    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:51:50.800212 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800188    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:51:50.800212 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800193    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:51:50.800212 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800204    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:51:50.800212 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800209    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:51:50.800340 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800223    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:51:50.800340 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800229    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:51:50.800340 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800234    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:51:50.800340 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800240    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:51:50.800340 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800255    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:51:50.800340 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.800262    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:51:50.801804 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.801794    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:51:50.801843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.801804    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:51:50.804455 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.804426    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:51:50.804455 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.804425    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:51:50.804586 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.804550    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-40.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:51:50.805712 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.805696    2580 server.go:1268] "Started kubelet"
Sep 09 20:51:50.806023 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.805829    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:51:50.806144 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.805823    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:51:50.806392 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.806369    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:51:50.806734 ip-10-0-130-40 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:51:50.807713 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.807606    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:51:50.809159 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.809145    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:51:50.812580 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.812559    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:51:50.813367 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.813345    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:51:50.813820 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.813802    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:51:50.813920 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.813903    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:50.814530 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.813666    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-40.ec2.internal.18d3c24c7f52c16f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-40.ec2.internal,UID:ip-10-0-130-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-40.ec2.internal,},FirstTimestamp:2026-09-09 20:51:50.805651823 +0000 UTC m=+0.366679130,LastTimestamp:2026-09-09 20:51:50.805651823 +0000 UTC m=+0.366679130,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-40.ec2.internal,}"
Sep 09 20:51:50.814636 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814616    2580 factory.go:55] Registering systemd factory
Sep 09 20:51:50.814636 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814628    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:51:50.814720 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814645    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:51:50.814720 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814655    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:51:50.814720 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814643    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:51:50.814836 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814735    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:51:50.814836 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814746    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:51:50.814905 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814882    2580 factory.go:153] Registering CRI-O factory
Sep 09 20:51:50.814905 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814892    2580 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:51:50.814981 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814929    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:51:50.814981 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814943    2580 factory.go:103] Registering Raw factory
Sep 09 20:51:50.814981 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.814957    2580 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:51:50.815329 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.815316    2580 manager.go:349] Starting recovery of all containers
Sep 09 20:51:50.821124 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.821089    2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-40.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:51:50.821208 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.821119    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:51:50.826451 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.826433    2580 manager.go:354] Recovery completed
Sep 09 20:51:50.827727 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.827708    2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 09 20:51:50.830506 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.830494    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:51:50.832162 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.832142    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:51:50.832227 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.832181    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:51:50.832227 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.832196    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:51:50.832795 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.832744    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:51:50.832845 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.832745    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:51:50.832845 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.832834    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:51:50.832911 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.832851    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:51:50.834278 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.834262    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:51:50.834322 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.834288    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:51:50.834322 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.834298    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:51:50.834675 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.834615    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-40.ec2.internal.18d3c24c80e748c2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-40.ec2.internal,UID:ip-10-0-130-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-40.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-40.ec2.internal,},FirstTimestamp:2026-09-09 20:51:50.83216301 +0000 UTC m=+0.393190319,LastTimestamp:2026-09-09 20:51:50.83216301 +0000 UTC m=+0.393190319,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-40.ec2.internal,}"
Sep 09 20:51:50.834923 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.834912    2580 policy_none.go:49] "None policy: Start"
Sep 09 20:51:50.834972 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.834927    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:51:50.834972 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.834937    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:51:50.836890 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.836878    2580 policy_none.go:47] "Start"
Sep 09 20:51:50.843332 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.843179    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-40.ec2.internal.18d3c24c80e7aa41  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-40.ec2.internal,UID:ip-10-0-130-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-40.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-40.ec2.internal,},FirstTimestamp:2026-09-09 20:51:50.832187969 +0000 UTC m=+0.393215276,LastTimestamp:2026-09-09 20:51:50.832187969 +0000 UTC m=+0.393215276,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-40.ec2.internal,}"
Sep 09 20:51:50.850237 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.850221    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4bz28"
Sep 09 20:51:50.855604 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.855545    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-40.ec2.internal.18d3c24c80e7de96  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-40.ec2.internal,UID:ip-10-0-130-40.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-40.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-40.ec2.internal,},FirstTimestamp:2026-09-09 20:51:50.832201366 +0000 UTC m=+0.393228675,LastTimestamp:2026-09-09 20:51:50.832201366 +0000 UTC m=+0.393228675,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-40.ec2.internal,}"
Sep 09 20:51:50.858884 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.858867    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4bz28"
Sep 09 20:51:50.875340 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.875327    2580 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:51:50.901586 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.875371    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:51:50.901586 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.875384    2580 server.go:87] "Starting device plugin registration server"
Sep 09 20:51:50.901586 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.875667    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:51:50.901586 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.875676    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:51:50.901586 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.875818    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:51:50.901586 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.875934    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:51:50.901586 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.875967    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:51:50.901586 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.876440    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:51:50.901586 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.876493    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:50.927903 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.927876    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:51:50.929233 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.929215    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:51:50.929291 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.929273    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:51:50.929339 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.929303    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:51:50.929410 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.929394    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 20:51:50.932003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.931983    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:51:50.976564 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.976541    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:51:50.977470 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.977453    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:51:50.977592 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.977483    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:51:50.977592 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.977493    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:51:50.977592 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.977515    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-40.ec2.internal"
Sep 09 20:51:50.986717 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:50.986650    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-40.ec2.internal"
Sep 09 20:51:50.986717 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:50.986672    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-40.ec2.internal\": node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:51.014612 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.014591    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:51.029630 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.029606    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal"]
Sep 09 20:51:51.029687 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.029664    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:51:51.030507 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.030493    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:51:51.030579 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.030518    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:51:51.030579 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.030528    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:51:51.032685 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.032672    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:51:51.032810 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.032795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.032866 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.032826    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:51:51.033429 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.033414    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:51:51.033498 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.033434    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:51:51.033498 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.033462    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:51:51.033498 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.033472    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:51:51.033613 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.033441    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:51:51.033613 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.033523    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:51:51.035599 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.035584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.035677 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.035606    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:51:51.036242 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.036226    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:51:51.036331 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.036249    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:51:51.036331 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.036262    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:51:51.050255 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.050235    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-40.ec2.internal\" not found" node="ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.054591 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.054575    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-40.ec2.internal\" not found" node="ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.114954 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.114917    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:51.117105 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.117086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f5109ee7be5c275f23b5ff16d17c015-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal\" (UID: \"5f5109ee7be5c275f23b5ff16d17c015\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.117168 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.117112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/16a910e8d25a6819bae738e4b0e06e46-config\") pod \"kube-apiserver-proxy-ip-10-0-130-40.ec2.internal\" (UID: \"16a910e8d25a6819bae738e4b0e06e46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.117168 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.117138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5f5109ee7be5c275f23b5ff16d17c015-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal\" (UID: \"5f5109ee7be5c275f23b5ff16d17c015\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.215436 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.215391    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:51.217612 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.217594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f5109ee7be5c275f23b5ff16d17c015-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal\" (UID: \"5f5109ee7be5c275f23b5ff16d17c015\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.217668 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.217621    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/16a910e8d25a6819bae738e4b0e06e46-config\") pod \"kube-apiserver-proxy-ip-10-0-130-40.ec2.internal\" (UID: \"16a910e8d25a6819bae738e4b0e06e46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.217668 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.217638    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5f5109ee7be5c275f23b5ff16d17c015-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal\" (UID: \"5f5109ee7be5c275f23b5ff16d17c015\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.217782 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.217677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5f5109ee7be5c275f23b5ff16d17c015-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal\" (UID: \"5f5109ee7be5c275f23b5ff16d17c015\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.217782 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.217709    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f5109ee7be5c275f23b5ff16d17c015-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal\" (UID: \"5f5109ee7be5c275f23b5ff16d17c015\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.217782 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.217731    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/16a910e8d25a6819bae738e4b0e06e46-config\") pod \"kube-apiserver-proxy-ip-10-0-130-40.ec2.internal\" (UID: \"16a910e8d25a6819bae738e4b0e06e46\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.316171 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.316096    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:51.352383 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.352364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.358248 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.358228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.416906 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.416874    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:51.517445 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.517415    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:51.617929 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.617892    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:51.718497 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:51.718460    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-40.ec2.internal\" not found"
Sep 09 20:51:51.758794 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.758747    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:51:51.759286 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.758905    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:51:51.816994 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.816966    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:51:51.861579 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.861539    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:46:51 +0000 UTC" deadline="2028-06-12 12:56:27.952735678 +0000 UTC"
Sep 09 20:51:51.861579 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.861570    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15400h4m36.091167898s"
Sep 09 20:51:51.914992 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.914926    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.927059 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.927039    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:51:51.928668 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.928654    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal"
Sep 09 20:51:51.937034 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:51.937014    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:51:52.056386 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.056353    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:51:52.075212 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:52.075185    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5109ee7be5c275f23b5ff16d17c015.slice/crio-f670f1f8f7fa76f5a70516284d64d0b1860ff37b388f95a41bead82f5f1b8753 WatchSource:0}: Error finding container f670f1f8f7fa76f5a70516284d64d0b1860ff37b388f95a41bead82f5f1b8753: Status 404 returned error can't find the container with id f670f1f8f7fa76f5a70516284d64d0b1860ff37b388f95a41bead82f5f1b8753
Sep 09 20:51:52.075450 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:52.075430    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a910e8d25a6819bae738e4b0e06e46.slice/crio-b0f71f4dcf36ed427bb97bdc9ce500759f752b592a91cc323171a3ae4e75fd0c WatchSource:0}: Error finding container b0f71f4dcf36ed427bb97bdc9ce500759f752b592a91cc323171a3ae4e75fd0c: Status 404 returned error can't find the container with id b0f71f4dcf36ed427bb97bdc9ce500759f752b592a91cc323171a3ae4e75fd0c
Sep 09 20:51:52.080755 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.080739    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:51:52.118569 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.117247    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:51:52.685884 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.685850    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:51:52.797157 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.797128    2580 apiserver.go:52] "Watching apiserver"
Sep 09 20:51:52.807313 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.807287    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:51:52.807782 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.807749    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal","openshift-multus/multus-additional-cni-plugins-6phgb","openshift-multus/network-metrics-daemon-nzvfm","openshift-network-operator/iptables-alerter-ngnpn","openshift-ovn-kubernetes/ovnkube-node-fc6xb","kube-system/global-pull-secret-syncer-zdt9w","kube-system/konnectivity-agent-q2tpw","kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr","openshift-cluster-node-tuning-operator/tuned-284xd","openshift-dns/node-resolver-xxnbh","openshift-multus/multus-wh7mr","openshift-network-diagnostics/network-check-target-xwlqv","openshift-image-registry/node-ca-7zwsv"]
Sep 09 20:51:52.810821 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.810780    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:51:52.815499 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.814523    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jp6cg\""
Sep 09 20:51:52.815499 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.814789    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:51:52.815499 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.814998    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:51:52.815866 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.815825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.816869 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.816219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:52.816869 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.816271    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:51:52.816869 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:52.816285    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:51:52.817875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.817853    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:52.818889 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.818473    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:51:52.818889 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.818509    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f879h\""
Sep 09 20:51:52.818889 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.818728    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:51:52.820261 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.820239    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:51:52.820531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.820512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:51:52.820753 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.820740    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h8wzh\""
Sep 09 20:51:52.820967 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.820951    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:51:52.821487 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.821468    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.823353 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.823050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.823353 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.823314    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:51:52.825933 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.825293    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w64nt\""
Sep 09 20:51:52.825933 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.825451    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:51:52.825933 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.825722    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:52.826912 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.826831    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:51:52.827066 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.827047    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:51:52.827294 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.827276    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:51:52.827472 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.827456    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:51:52.827620 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.827603    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:51:52.827795 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.827779    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:51:52.827991 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.827974    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:51:52.828220 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828200    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9qfbx\""
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/334cd7af-26a0-4b78-8683-bdfae8ae703e-iptables-alerter-script\") pod \"iptables-alerter-ngnpn\" (UID: \"334cd7af-26a0-4b78-8683-bdfae8ae703e\") " pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/334cd7af-26a0-4b78-8683-bdfae8ae703e-host-slash\") pod \"iptables-alerter-ngnpn\" (UID: \"334cd7af-26a0-4b78-8683-bdfae8ae703e\") " pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2fr\" (UniqueName: \"kubernetes.io/projected/334cd7af-26a0-4b78-8683-bdfae8ae703e-kube-api-access-cg2fr\") pod \"iptables-alerter-ngnpn\" (UID: \"334cd7af-26a0-4b78-8683-bdfae8ae703e\") " pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4df75668-d74f-4dde-81dd-ecac0a82140a-hosts-file\") pod \"node-resolver-xxnbh\" (UID: \"4df75668-d74f-4dde-81dd-ecac0a82140a\") " pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828722    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g92vj\" (UniqueName: \"kubernetes.io/projected/4df75668-d74f-4dde-81dd-ecac0a82140a-kube-api-access-g92vj\") pod \"node-resolver-xxnbh\" (UID: \"4df75668-d74f-4dde-81dd-ecac0a82140a\") " pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk6hd\" (UniqueName: \"kubernetes.io/projected/b7b1474e-02e4-4041-a48a-2a3a652388c9-kube-api-access-hk6hd\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/596b15b6-2d26-46db-a293-495852388c4f-agent-certs\") pod \"konnectivity-agent-q2tpw\" (UID: \"596b15b6-2d26-46db-a293-495852388c4f\") " pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/596b15b6-2d26-46db-a293-495852388c4f-konnectivity-ca\") pod \"konnectivity-agent-q2tpw\" (UID: \"596b15b6-2d26-46db-a293-495852388c4f\") " pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df75668-d74f-4dde-81dd-ecac0a82140a-tmp-dir\") pod \"node-resolver-xxnbh\" (UID: \"4df75668-d74f-4dde-81dd-ecac0a82140a\") " pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.828904    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.829324    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.829378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.829325    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-qw2pz\""
Sep 09 20:51:52.829711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.829681    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:51:52.830565 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.830233    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:51:52.830996 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.830974    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:51:52.831358 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.831332    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5t8x\""
Sep 09 20:51:52.831617 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.831599    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:52.831687 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.831631    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:51:52.834257 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.833899    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:51:52.834257 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:52.833966    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:51:52.836984 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.836236    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:51:52.836984 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.836327    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:51:52.836984 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.836790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rkprc\""
Sep 09 20:51:52.836984 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.836934    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:52.837510 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:52.837013    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:51:52.840211 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.840187    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:52.842595 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.842573    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7shsv\""
Sep 09 20:51:52.842833 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.842815    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:51:52.843644 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.843625    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:51:52.844626 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.844264    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:51:52.856009 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.855991    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4zk2m"
Sep 09 20:51:52.867504 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.867490    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4zk2m"
Sep 09 20:51:52.915631 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.915609    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:51:52.929273 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929244    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-cni-binary-copy\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.929379 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4df75668-d74f-4dde-81dd-ecac0a82140a-hosts-file\") pod \"node-resolver-xxnbh\" (UID: \"4df75668-d74f-4dde-81dd-ecac0a82140a\") " pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.929379 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929330    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/374b7dce-d3a8-4665-a0ad-96c74a838738-dbus\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:52.929379 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929357    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b73b24e-5ef0-4299-90d8-b7932046333b-serviceca\") pod \"node-ca-7zwsv\" (UID: \"9b73b24e-5ef0-4299-90d8-b7932046333b\") " pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:52.929563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929385    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-run-ovn-kubernetes\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.929563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929411    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/326bde36-e763-476f-92d8-e411d84cad01-ovnkube-config\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.929563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8wr\" (UniqueName: \"kubernetes.io/projected/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-kube-api-access-cx8wr\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:52.929563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929471    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-modprobe-d\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.929563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-run-netns\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.929563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/334cd7af-26a0-4b78-8683-bdfae8ae703e-host-slash\") pod \"iptables-alerter-ngnpn\" (UID: \"334cd7af-26a0-4b78-8683-bdfae8ae703e\") " pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:52.929563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929550    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g92vj\" (UniqueName: \"kubernetes.io/projected/4df75668-d74f-4dde-81dd-ecac0a82140a-kube-api-access-g92vj\") pod \"node-resolver-xxnbh\" (UID: \"4df75668-d74f-4dde-81dd-ecac0a82140a\") " pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-cnibin\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-etc-kubernetes\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929630    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-etc-selinux\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929654    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-slash\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929681    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk6hd\" (UniqueName: \"kubernetes.io/projected/b7b1474e-02e4-4041-a48a-2a3a652388c9-kube-api-access-hk6hd\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929709    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/596b15b6-2d26-46db-a293-495852388c4f-konnectivity-ca\") pod \"konnectivity-agent-q2tpw\" (UID: \"596b15b6-2d26-46db-a293-495852388c4f\") " pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929736    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-cnibin\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929790    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-sys\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929841    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-log-socket\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.930385 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.929921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/334cd7af-26a0-4b78-8683-bdfae8ae703e-host-slash\") pod \"iptables-alerter-ngnpn\" (UID: \"334cd7af-26a0-4b78-8683-bdfae8ae703e\") " pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:52.930385 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930015    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4df75668-d74f-4dde-81dd-ecac0a82140a-hosts-file\") pod \"node-resolver-xxnbh\" (UID: \"4df75668-d74f-4dde-81dd-ecac0a82140a\") " pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.930622 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930596    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:52.930689 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930650    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-registration-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:52.930689 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdth\" (UniqueName: \"kubernetes.io/projected/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-kube-api-access-8zdth\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:52.930804 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:52.930804 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930741    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-run-systemd\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.930804 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930765    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/326bde36-e763-476f-92d8-e411d84cad01-ovnkube-script-lib\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.930978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjpbq\" (UniqueName: \"kubernetes.io/projected/326bde36-e763-476f-92d8-e411d84cad01-kube-api-access-zjpbq\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.930978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-daemon-config\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.930978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930861    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-var-lib-openvswitch\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.930978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930885    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-etc-openvswitch\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.930978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930909    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-cni-bin\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.930978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8054739c-ef5e-4bfd-83ab-797a962f6b1b-tmp\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.930978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df75668-d74f-4dde-81dd-ecac0a82140a-tmp-dir\") pod \"node-resolver-xxnbh\" (UID: \"4df75668-d74f-4dde-81dd-ecac0a82140a\") " pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.930986    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-hostroot\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-conf-dir\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931079    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlsz\" (UniqueName: \"kubernetes.io/projected/9b73b24e-5ef0-4299-90d8-b7932046333b-kube-api-access-qjlsz\") pod \"node-ca-7zwsv\" (UID: \"9b73b24e-5ef0-4299-90d8-b7932046333b\") " pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/374b7dce-d3a8-4665-a0ad-96c74a838738-kubelet-config\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931129    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-cni-dir\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-device-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931181    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-kubelet\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931206    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-run-openvswitch\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931231    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-var-lib-kubelet\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/596b15b6-2d26-46db-a293-495852388c4f-konnectivity-ca\") pod \"konnectivity-agent-q2tpw\" (UID: \"596b15b6-2d26-46db-a293-495852388c4f\") " pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931284    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/326bde36-e763-476f-92d8-e411d84cad01-ovn-node-metrics-cert\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-sysconfig\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.931337 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931339    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-systemd\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931382    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/334cd7af-26a0-4b78-8683-bdfae8ae703e-iptables-alerter-script\") pod \"iptables-alerter-ngnpn\" (UID: \"334cd7af-26a0-4b78-8683-bdfae8ae703e\") " pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931408    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-run-netns\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931434    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-var-lib-kubelet\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931548    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4r7p\" (UniqueName: \"kubernetes.io/projected/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-kube-api-access-j4r7p\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-systemd-units\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-sysctl-conf\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931627    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-system-cni-dir\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931656    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931680    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-host\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b73b24e-5ef0-4299-90d8-b7932046333b-host\") pod \"node-ca-7zwsv\" (UID: \"9b73b24e-5ef0-4299-90d8-b7932046333b\") " pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-var-lib-cni-bin\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-var-lib-cni-multus\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931760    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931779    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2fr\" (UniqueName: \"kubernetes.io/projected/334cd7af-26a0-4b78-8683-bdfae8ae703e-kube-api-access-cg2fr\") pod \"iptables-alerter-ngnpn\" (UID: \"334cd7af-26a0-4b78-8683-bdfae8ae703e\") " pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-run-multus-certs\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.932065 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931814    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-run-k8s-cni-cncf-io\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931936    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-socket-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.931971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-system-cni-dir\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932001    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-node-log\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/334cd7af-26a0-4b78-8683-bdfae8ae703e-iptables-alerter-script\") pod \"iptables-alerter-ngnpn\" (UID: \"334cd7af-26a0-4b78-8683-bdfae8ae703e\") " pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932056    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-cni-netd\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932085    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/326bde36-e763-476f-92d8-e411d84cad01-env-overrides\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932112    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-kubernetes\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-socket-dir-parent\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932167    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-sys-fs\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-run-ovn\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932232    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-lib-modules\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:52.932242    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932270    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-tuned\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4df75668-d74f-4dde-81dd-ecac0a82140a-tmp-dir\") pod \"node-resolver-xxnbh\" (UID: \"4df75668-d74f-4dde-81dd-ecac0a82140a\") " pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.932778 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:52.932324    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs podName:b7b1474e-02e4-4041-a48a-2a3a652388c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:51:53.432302197 +0000 UTC m=+2.993329496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs") pod "network-metrics-daemon-nzvfm" (UID: "b7b1474e-02e4-4041-a48a-2a3a652388c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:51:52.933603 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvhbc\" (UniqueName: \"kubernetes.io/projected/8054739c-ef5e-4bfd-83ab-797a962f6b1b-kube-api-access-pvhbc\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.933603 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/596b15b6-2d26-46db-a293-495852388c4f-agent-certs\") pod \"konnectivity-agent-q2tpw\" (UID: \"596b15b6-2d26-46db-a293-495852388c4f\") " pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:51:52.933603 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932533    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:52.933603 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932575    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-sysctl-d\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.933603 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.932602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-run\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:52.933603 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.933000    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:51:52.937893 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.937690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/596b15b6-2d26-46db-a293-495852388c4f-agent-certs\") pod \"konnectivity-agent-q2tpw\" (UID: \"596b15b6-2d26-46db-a293-495852388c4f\") " pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:51:52.938127 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.938072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal" event={"ID":"16a910e8d25a6819bae738e4b0e06e46","Type":"ContainerStarted","Data":"b0f71f4dcf36ed427bb97bdc9ce500759f752b592a91cc323171a3ae4e75fd0c"}
Sep 09 20:51:52.941978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.941799    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal" event={"ID":"5f5109ee7be5c275f23b5ff16d17c015","Type":"ContainerStarted","Data":"f670f1f8f7fa76f5a70516284d64d0b1860ff37b388f95a41bead82f5f1b8753"}
Sep 09 20:51:52.941978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.941872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g92vj\" (UniqueName: \"kubernetes.io/projected/4df75668-d74f-4dde-81dd-ecac0a82140a-kube-api-access-g92vj\") pod \"node-resolver-xxnbh\" (UID: \"4df75668-d74f-4dde-81dd-ecac0a82140a\") " pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:52.942126 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.942094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk6hd\" (UniqueName: \"kubernetes.io/projected/b7b1474e-02e4-4041-a48a-2a3a652388c9-kube-api-access-hk6hd\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:52.944449 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:52.944423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2fr\" (UniqueName: \"kubernetes.io/projected/334cd7af-26a0-4b78-8683-bdfae8ae703e-kube-api-access-cg2fr\") pod \"iptables-alerter-ngnpn\" (UID: \"334cd7af-26a0-4b78-8683-bdfae8ae703e\") " pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:53.033310 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/326bde36-e763-476f-92d8-e411d84cad01-ovn-node-metrics-cert\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.033310 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-sysconfig\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-systemd\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-run-netns\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-var-lib-kubelet\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4r7p\" (UniqueName: \"kubernetes.io/projected/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-kube-api-access-j4r7p\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033402    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-systemd-units\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-sysctl-conf\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033432    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-system-cni-dir\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033474    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-host\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b73b24e-5ef0-4299-90d8-b7932046333b-host\") pod \"node-ca-7zwsv\" (UID: \"9b73b24e-5ef0-4299-90d8-b7932046333b\") " pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-var-lib-cni-bin\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.033531 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-var-lib-cni-multus\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033554    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-run-multus-certs\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033585    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-run-k8s-cni-cncf-io\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033603    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-socket-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033623    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-system-cni-dir\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-node-log\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-cni-netd\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033682    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/326bde36-e763-476f-92d8-e411d84cad01-env-overrides\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-kubernetes\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033712    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-socket-dir-parent\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-sys-fs\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-run-ovn\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-lib-modules\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-tuned\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvhbc\" (UniqueName: \"kubernetes.io/projected/8054739c-ef5e-4bfd-83ab-797a962f6b1b-kube-api-access-pvhbc\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.034062 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033863    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-sysctl-d\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-run\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-cni-binary-copy\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033917    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/374b7dce-d3a8-4665-a0ad-96c74a838738-dbus\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033942    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b73b24e-5ef0-4299-90d8-b7932046333b-serviceca\") pod \"node-ca-7zwsv\" (UID: \"9b73b24e-5ef0-4299-90d8-b7932046333b\") " pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-run-ovn-kubernetes\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.033987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/326bde36-e763-476f-92d8-e411d84cad01-ovnkube-config\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034002    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8wr\" (UniqueName: \"kubernetes.io/projected/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-kube-api-access-cx8wr\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034018    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-modprobe-d\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-run-netns\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-cnibin\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034077    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-etc-kubernetes\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-etc-selinux\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-slash\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034181    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-cnibin\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-sys\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.034802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034213    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-log-socket\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034254    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-registration-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdth\" (UniqueName: \"kubernetes.io/projected/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-kube-api-access-8zdth\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034290    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034306    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-run-systemd\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/326bde36-e763-476f-92d8-e411d84cad01-ovnkube-script-lib\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjpbq\" (UniqueName: \"kubernetes.io/projected/326bde36-e763-476f-92d8-e411d84cad01-kube-api-access-zjpbq\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-daemon-config\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034386    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-var-lib-openvswitch\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-etc-openvswitch\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-cni-bin\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034455    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8054739c-ef5e-4bfd-83ab-797a962f6b1b-tmp\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-hostroot\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034487    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-conf-dir\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlsz\" (UniqueName: \"kubernetes.io/projected/9b73b24e-5ef0-4299-90d8-b7932046333b-kube-api-access-qjlsz\") pod \"node-ca-7zwsv\" (UID: \"9b73b24e-5ef0-4299-90d8-b7932046333b\") " pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:53.035518 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/374b7dce-d3a8-4665-a0ad-96c74a838738-kubelet-config\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034596    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-cni-dir\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-device-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-kubelet\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034685    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-run-openvswitch\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034713    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-var-lib-kubelet\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.034741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.035469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-run-ovn-kubernetes\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.035529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-sysconfig\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.035562    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-systemd\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.035600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-run-netns\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.035634    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-var-lib-kubelet\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.035866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-systemd-units\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036035    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-sysctl-conf\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-system-cni-dir\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-host\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036284    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b73b24e-5ef0-4299-90d8-b7932046333b-host\") pod \"node-ca-7zwsv\" (UID: \"9b73b24e-5ef0-4299-90d8-b7932046333b\") " pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:53.036529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036320    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-var-lib-cni-bin\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036337    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/326bde36-e763-476f-92d8-e411d84cad01-ovnkube-config\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036358    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-var-lib-cni-multus\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036398    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-run-multus-certs\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036451    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-modprobe-d\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-run-netns\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-cnibin\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036574    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-etc-kubernetes\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036644    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-etc-selinux\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-slash\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036749    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-cnibin\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-sys\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.036910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-log-socket\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.036991    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.037050    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret podName:374b7dce-d3a8-4665-a0ad-96c74a838738 nodeName:}" failed. No retries permitted until 2026-09-09 20:51:53.537030376 +0000 UTC m=+3.098057695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret") pod "global-pull-secret-syncer-zdt9w" (UID: "374b7dce-d3a8-4665-a0ad-96c74a838738") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.037290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-hostroot\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.037455 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.037344    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-conf-dir\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.037506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.037776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/374b7dce-d3a8-4665-a0ad-96c74a838738-kubelet-config\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.037867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-cni-dir\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.037882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.037925    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-device-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.037974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-kubelet\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.037989    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-run-openvswitch\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038020    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-var-lib-kubelet\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-run-ovn\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038132    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-lib-modules\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038175    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-host-run-k8s-cni-cncf-io\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038225    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/326bde36-e763-476f-92d8-e411d84cad01-ovn-node-metrics-cert\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.038319 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-socket-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.039006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038330    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-kubernetes\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.039006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038345    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-system-cni-dir\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.039006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-node-log\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.039006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-run\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.039006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038657    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/326bde36-e763-476f-92d8-e411d84cad01-env-overrides\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.039006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-cni-netd\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.039006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038784    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/374b7dce-d3a8-4665-a0ad-96c74a838738-dbus\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:53.039006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038917    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-cni-binary-copy\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.039006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.038987    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-socket-dir-parent\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.039462 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-sys-fs\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.039462 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-sysctl-d\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.039462 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039158    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.039462 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b73b24e-5ef0-4299-90d8-b7932046333b-serviceca\") pod \"node-ca-7zwsv\" (UID: \"9b73b24e-5ef0-4299-90d8-b7932046333b\") " pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:53.039462 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-var-lib-openvswitch\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.039462 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039290    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-etc-openvswitch\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.039764 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-host-cni-bin\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.039764 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/326bde36-e763-476f-92d8-e411d84cad01-ovnkube-script-lib\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.039764 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039743    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-multus-daemon-config\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.039930 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039813    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/326bde36-e763-476f-92d8-e411d84cad01-run-systemd\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.039930 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.039822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-registration-dir\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.040196 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.040177    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-cni-binary-copy\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.041494 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.041467    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:51:53.041494 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.041490    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:51:53.041494 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.041500    2580 projected.go:196] Error preparing data for projected volume kube-api-access-7bvgz for pod openshift-network-diagnostics/network-check-target-xwlqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:51:53.041746 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.041559    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz podName:45839b82-7bb9-4f27-8efe-89affa6fc06a nodeName:}" failed. No retries permitted until 2026-09-09 20:51:53.541546821 +0000 UTC m=+3.102574154 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7bvgz" (UniqueName: "kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz") pod "network-check-target-xwlqv" (UID: "45839b82-7bb9-4f27-8efe-89affa6fc06a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:51:53.042429 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.042333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8054739c-ef5e-4bfd-83ab-797a962f6b1b-tmp\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.042725 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.042482    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8054739c-ef5e-4bfd-83ab-797a962f6b1b-etc-tuned\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.044348 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.044324    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4r7p\" (UniqueName: \"kubernetes.io/projected/fe5b2e20-19a6-4742-8f06-c1f10f896bb4-kube-api-access-j4r7p\") pod \"multus-wh7mr\" (UID: \"fe5b2e20-19a6-4742-8f06-c1f10f896bb4\") " pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.046919 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.046894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8wr\" (UniqueName: \"kubernetes.io/projected/f3b6ca73-058e-4a22-9cfa-dbbf72f9564b-kube-api-access-cx8wr\") pod \"multus-additional-cni-plugins-6phgb\" (UID: \"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b\") " pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.047981 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.047954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlsz\" (UniqueName: \"kubernetes.io/projected/9b73b24e-5ef0-4299-90d8-b7932046333b-kube-api-access-qjlsz\") pod \"node-ca-7zwsv\" (UID: \"9b73b24e-5ef0-4299-90d8-b7932046333b\") " pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:53.048148 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.048122    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvhbc\" (UniqueName: \"kubernetes.io/projected/8054739c-ef5e-4bfd-83ab-797a962f6b1b-kube-api-access-pvhbc\") pod \"tuned-284xd\" (UID: \"8054739c-ef5e-4bfd-83ab-797a962f6b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.048767 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.048747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjpbq\" (UniqueName: \"kubernetes.io/projected/326bde36-e763-476f-92d8-e411d84cad01-kube-api-access-zjpbq\") pod \"ovnkube-node-fc6xb\" (UID: \"326bde36-e763-476f-92d8-e411d84cad01\") " pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.049630 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.049597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdth\" (UniqueName: \"kubernetes.io/projected/7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e-kube-api-access-8zdth\") pod \"aws-ebs-csi-driver-node-7tvgr\" (UID: \"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.126134 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.126099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:51:53.141701 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.141666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xxnbh"
Sep 09 20:51:53.152284 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.152262    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ngnpn"
Sep 09 20:51:53.159637 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.159610    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:51:53.166731 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.166708    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wh7mr"
Sep 09 20:51:53.175959 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.175942    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr"
Sep 09 20:51:53.183861 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.183834    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-284xd"
Sep 09 20:51:53.193794 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.193740    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6phgb"
Sep 09 20:51:53.200703 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.200681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7zwsv"
Sep 09 20:51:53.438149 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.438117    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:53.438340 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.438302    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:51:53.438424 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.438390    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs podName:b7b1474e-02e4-4041-a48a-2a3a652388c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:51:54.438366829 +0000 UTC m=+3.999394147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs") pod "network-metrics-daemon-nzvfm" (UID: "b7b1474e-02e4-4041-a48a-2a3a652388c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:51:53.539341 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.539242    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:53.539497 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.539429    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:51:53.539553 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.539520    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret podName:374b7dce-d3a8-4665-a0ad-96c74a838738 nodeName:}" failed. No retries permitted until 2026-09-09 20:51:54.539496709 +0000 UTC m=+4.100524022 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret") pod "global-pull-secret-syncer-zdt9w" (UID: "374b7dce-d3a8-4665-a0ad-96c74a838738") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:51:53.640420 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.640379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:51:53.640578 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.640532    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:51:53.640578 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.640549    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:51:53.640578 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.640558    2580 projected.go:196] Error preparing data for projected volume kube-api-access-7bvgz for pod openshift-network-diagnostics/network-check-target-xwlqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:51:53.640735 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:53.640622    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz podName:45839b82-7bb9-4f27-8efe-89affa6fc06a nodeName:}" failed. No retries permitted until 2026-09-09 20:51:54.640603834 +0000 UTC m=+4.201631146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bvgz" (UniqueName: "kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz") pod "network-check-target-xwlqv" (UID: "45839b82-7bb9-4f27-8efe-89affa6fc06a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:51:53.872334 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.872293    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:46:53 +0000 UTC" deadline="2028-03-10 03:16:06.468204856 +0000 UTC"
Sep 09 20:51:53.872334 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:53.872330    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13134h24m12.595878737s"
Sep 09 20:51:54.233115 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:54.233082    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8054739c_ef5e_4bfd_83ab_797a962f6b1b.slice/crio-9df2d7f5c5f334e82efc0fa575489f04f3180b7df33b128ccd67eb812c8e8d3a WatchSource:0}: Error finding container 9df2d7f5c5f334e82efc0fa575489f04f3180b7df33b128ccd67eb812c8e8d3a: Status 404 returned error can't find the container with id 9df2d7f5c5f334e82efc0fa575489f04f3180b7df33b128ccd67eb812c8e8d3a
Sep 09 20:51:54.235094 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:54.235069    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b73b24e_5ef0_4299_90d8_b7932046333b.slice/crio-6858c01440c978dffc8244fa9dd857e0c7462342534c0c9f74c10bb99851174d WatchSource:0}: Error finding container 6858c01440c978dffc8244fa9dd857e0c7462342534c0c9f74c10bb99851174d: Status 404 returned error can't find the container with id 6858c01440c978dffc8244fa9dd857e0c7462342534c0c9f74c10bb99851174d
Sep 09 20:51:54.237065 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:54.236955    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334cd7af_26a0_4b78_8683_bdfae8ae703e.slice/crio-8c6f7797ed6b6a0b8032b5af5f195538608a195962a976b60d93de0144cd82b2 WatchSource:0}: Error finding container 8c6f7797ed6b6a0b8032b5af5f195538608a195962a976b60d93de0144cd82b2: Status 404 returned error can't find the container with id 8c6f7797ed6b6a0b8032b5af5f195538608a195962a976b60d93de0144cd82b2
Sep 09 20:51:54.240427 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:54.240403    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5b2e20_19a6_4742_8f06_c1f10f896bb4.slice/crio-f09b7037583f54d6c3a5b2c09d46348ee6aa868428ae6bc05845161e50542b61 WatchSource:0}: Error finding container f09b7037583f54d6c3a5b2c09d46348ee6aa868428ae6bc05845161e50542b61: Status 404 returned error can't find the container with id f09b7037583f54d6c3a5b2c09d46348ee6aa868428ae6bc05845161e50542b61
Sep 09 20:51:54.241568 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:54.241495    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4df75668_d74f_4dde_81dd_ecac0a82140a.slice/crio-3906b0abcba410ed056a47aeeb6df973047cd4bb44a5069cf070ad4c3de2e618 WatchSource:0}: Error finding container 3906b0abcba410ed056a47aeeb6df973047cd4bb44a5069cf070ad4c3de2e618: Status 404 returned error can't find the container with id 3906b0abcba410ed056a47aeeb6df973047cd4bb44a5069cf070ad4c3de2e618
Sep 09 20:51:54.242016 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:54.241891    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b6ca73_058e_4a22_9cfa_dbbf72f9564b.slice/crio-6ee3f886042c2d4fba233e4d0ee77e908b804f5e9e79ee25ff67c42343740435 WatchSource:0}: Error finding container 6ee3f886042c2d4fba233e4d0ee77e908b804f5e9e79ee25ff67c42343740435: Status 404 returned error can't find the container with id 6ee3f886042c2d4fba233e4d0ee77e908b804f5e9e79ee25ff67c42343740435
Sep 09 20:51:54.243062 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:51:54.243042    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab5d0cb_4e01_4fb5_96ff_acdd834c6d6e.slice/crio-be2c99cce3aac0a77d43dac52c05c4457906c6a526cb8d610f74627a4a261b0e WatchSource:0}: Error finding container be2c99cce3aac0a77d43dac52c05c4457906c6a526cb8d610f74627a4a261b0e: Status 404 returned error can't find the container with id be2c99cce3aac0a77d43dac52c05c4457906c6a526cb8d610f74627a4a261b0e
Sep 09 20:51:54.448660 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.448477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:54.448868 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.448632    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:51:54.448868 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.448767    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs podName:b7b1474e-02e4-4041-a48a-2a3a652388c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:51:56.448750459 +0000 UTC m=+6.009777755 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs") pod "network-metrics-daemon-nzvfm" (UID: "b7b1474e-02e4-4041-a48a-2a3a652388c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:51:54.549496 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.549466    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:54.549620 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.549604    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:51:54.549684 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.549673    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret podName:374b7dce-d3a8-4665-a0ad-96c74a838738 nodeName:}" failed. No retries permitted until 2026-09-09 20:51:56.549652625 +0000 UTC m=+6.110679932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret") pod "global-pull-secret-syncer-zdt9w" (UID: "374b7dce-d3a8-4665-a0ad-96c74a838738") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:51:54.650221 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.650179    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:51:54.650368 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.650338    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:51:54.650368 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.650362    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:51:54.650439 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.650376    2580 projected.go:196] Error preparing data for projected volume kube-api-access-7bvgz for pod openshift-network-diagnostics/network-check-target-xwlqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:51:54.650439 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.650437    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz podName:45839b82-7bb9-4f27-8efe-89affa6fc06a nodeName:}" failed. No retries permitted until 2026-09-09 20:51:56.65042276 +0000 UTC m=+6.211450062 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bvgz" (UniqueName: "kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz") pod "network-check-target-xwlqv" (UID: "45839b82-7bb9-4f27-8efe-89affa6fc06a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:51:54.873430 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.873364    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:46:53 +0000 UTC" deadline="2028-06-05 22:02:29.369983084 +0000 UTC"
Sep 09 20:51:54.873430 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.873410    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15241h10m34.496575296s"
Sep 09 20:51:54.932897 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.932858    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:54.933070 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.933011    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:51:54.934146 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.933434    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:51:54.934146 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.933531    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:51:54.934146 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.933590    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:54.934146 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:54.933664    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:51:54.950084 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.950051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q2tpw" event={"ID":"596b15b6-2d26-46db-a293-495852388c4f","Type":"ContainerStarted","Data":"f467b12683449ffc6a0eb33b15265b3d7cd269e9df0f72597e94c11250d5c4bf"}
Sep 09 20:51:54.953356 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.953325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerStarted","Data":"39d62d8e420de83eddb9b4c8bbebbb2f49faaddbe170772bc2b1993dbc875796"}
Sep 09 20:51:54.959054 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.959021    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerStarted","Data":"6ee3f886042c2d4fba233e4d0ee77e908b804f5e9e79ee25ff67c42343740435"}
Sep 09 20:51:54.960643 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.960613    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wh7mr" event={"ID":"fe5b2e20-19a6-4742-8f06-c1f10f896bb4","Type":"ContainerStarted","Data":"f09b7037583f54d6c3a5b2c09d46348ee6aa868428ae6bc05845161e50542b61"}
Sep 09 20:51:54.968570 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.968539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ngnpn" event={"ID":"334cd7af-26a0-4b78-8683-bdfae8ae703e","Type":"ContainerStarted","Data":"8c6f7797ed6b6a0b8032b5af5f195538608a195962a976b60d93de0144cd82b2"}
Sep 09 20:51:54.977912 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.977873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7zwsv" event={"ID":"9b73b24e-5ef0-4299-90d8-b7932046333b","Type":"ContainerStarted","Data":"6858c01440c978dffc8244fa9dd857e0c7462342534c0c9f74c10bb99851174d"}
Sep 09 20:51:54.984439 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.984412    2580 generic.go:356] "Generic (PLEG): container finished" podID="5f5109ee7be5c275f23b5ff16d17c015" containerID="36669435bb5285d37014edcd8ff3f51f9aafbc7d5b5fc36d2eaa66e7fcc51d38" exitCode=0
Sep 09 20:51:54.984547 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.984509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal" event={"ID":"5f5109ee7be5c275f23b5ff16d17c015","Type":"ContainerDied","Data":"36669435bb5285d37014edcd8ff3f51f9aafbc7d5b5fc36d2eaa66e7fcc51d38"}
Sep 09 20:51:54.992128 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.992100    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr" event={"ID":"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e","Type":"ContainerStarted","Data":"be2c99cce3aac0a77d43dac52c05c4457906c6a526cb8d610f74627a4a261b0e"}
Sep 09 20:51:54.997112 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:54.997086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xxnbh" event={"ID":"4df75668-d74f-4dde-81dd-ecac0a82140a","Type":"ContainerStarted","Data":"3906b0abcba410ed056a47aeeb6df973047cd4bb44a5069cf070ad4c3de2e618"}
Sep 09 20:51:55.000158 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:55.000086    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-284xd" event={"ID":"8054739c-ef5e-4bfd-83ab-797a962f6b1b","Type":"ContainerStarted","Data":"9df2d7f5c5f334e82efc0fa575489f04f3180b7df33b128ccd67eb812c8e8d3a"}
Sep 09 20:51:55.007302 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:55.006903    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal" event={"ID":"16a910e8d25a6819bae738e4b0e06e46","Type":"ContainerStarted","Data":"40722c65cb90d20db5b8c5876121c549a3cc24b6ff8944d658f8125457789af1"}
Sep 09 20:51:56.027967 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:56.027923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal" event={"ID":"5f5109ee7be5c275f23b5ff16d17c015","Type":"ContainerStarted","Data":"978d876c2677dfa460b4dcc57f12ee67d159b4b04675069a607ab915afff482e"}
Sep 09 20:51:56.047523 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:56.047207    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-40.ec2.internal" podStartSLOduration=4.047180517 podStartE2EDuration="4.047180517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:55.022287721 +0000 UTC m=+4.583315045" watchObservedRunningTime="2026-09-09 20:51:56.047180517 +0000 UTC m=+5.608207842"
Sep 09 20:51:56.466914 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:56.466517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:56.472126 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.472094    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:51:56.472326 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.472212    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs podName:b7b1474e-02e4-4041-a48a-2a3a652388c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:00.472186586 +0000 UTC m=+10.033213900 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs") pod "network-metrics-daemon-nzvfm" (UID: "b7b1474e-02e4-4041-a48a-2a3a652388c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:51:56.568506 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:56.567913    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:56.568506 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.568092    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:51:56.568506 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.568158    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret podName:374b7dce-d3a8-4665-a0ad-96c74a838738 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:00.568137373 +0000 UTC m=+10.129164675 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret") pod "global-pull-secret-syncer-zdt9w" (UID: "374b7dce-d3a8-4665-a0ad-96c74a838738") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:51:56.668617 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:56.668577    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:51:56.668874 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.668830    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:51:56.668874 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.668853    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:51:56.668874 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.668868    2580 projected.go:196] Error preparing data for projected volume kube-api-access-7bvgz for pod openshift-network-diagnostics/network-check-target-xwlqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:51:56.669076 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.668938    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz podName:45839b82-7bb9-4f27-8efe-89affa6fc06a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:00.668917357 +0000 UTC m=+10.229944672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bvgz" (UniqueName: "kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz") pod "network-check-target-xwlqv" (UID: "45839b82-7bb9-4f27-8efe-89affa6fc06a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:51:56.931072 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:56.930954    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:51:56.931282 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.931085    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:51:56.931627 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:56.931501    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:56.931627 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.931607    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:51:56.933576 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:56.933531    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:56.933690 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:56.933641    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:51:58.932217 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:58.932180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:51:58.932726 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:58.932339    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:51:58.932815 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:58.932798    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:51:58.932935 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:58.932912    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:51:58.932998 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:51:58.932923    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:51:58.933051 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:51:58.933014    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:00.499960 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:00.499919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:00.500339 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.500078    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:00.500339 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.500149    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs podName:b7b1474e-02e4-4041-a48a-2a3a652388c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:08.500129135 +0000 UTC m=+18.061156448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs") pod "network-metrics-daemon-nzvfm" (UID: "b7b1474e-02e4-4041-a48a-2a3a652388c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:00.600862 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:00.600387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:00.600862 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.600539    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:00.600862 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.600591    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret podName:374b7dce-d3a8-4665-a0ad-96c74a838738 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:08.600576209 +0000 UTC m=+18.161603507 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret") pod "global-pull-secret-syncer-zdt9w" (UID: "374b7dce-d3a8-4665-a0ad-96c74a838738") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:00.701851 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:00.701815    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:00.702028 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.701957    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:00.702028 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.701973    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:00.702028 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.701983    2580 projected.go:196] Error preparing data for projected volume kube-api-access-7bvgz for pod openshift-network-diagnostics/network-check-target-xwlqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:00.702140 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.702034    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz podName:45839b82-7bb9-4f27-8efe-89affa6fc06a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:08.702020223 +0000 UTC m=+18.263047519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bvgz" (UniqueName: "kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz") pod "network-check-target-xwlqv" (UID: "45839b82-7bb9-4f27-8efe-89affa6fc06a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:00.930038 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:00.929999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:00.930038 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:00.929999    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:00.931266 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.931222    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:00.931428 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.931339    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:00.931428 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:00.931347    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:00.931551 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:00.931461    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:02.933782 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:02.933737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:02.934257 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:02.933889    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:02.934257 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:02.934175    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:02.934341 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:02.934254    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:02.934341 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:02.934295    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:02.934426 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:02.934371    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:04.929640 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:04.929604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:04.930064 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:04.929604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:04.930064 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:04.929746    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:04.930064 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:04.929799    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:04.930064 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:04.929897    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:04.930064 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:04.929976    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:06.929883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:06.929847    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:06.930223 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:06.929975    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:06.930281 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:06.930273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:06.930354 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:06.930337    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:06.930392 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:06.930380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:06.930438 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:06.930426    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:08.563579 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:08.563488    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:08.563980 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.563675    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:08.563980 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.563750    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs podName:b7b1474e-02e4-4041-a48a-2a3a652388c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:24.563733297 +0000 UTC m=+34.124760595 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs") pod "network-metrics-daemon-nzvfm" (UID: "b7b1474e-02e4-4041-a48a-2a3a652388c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:08.664244 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:08.664201    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:08.664415 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.664384    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:08.664474 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.664458    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret podName:374b7dce-d3a8-4665-a0ad-96c74a838738 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:24.664436482 +0000 UTC m=+34.225463777 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret") pod "global-pull-secret-syncer-zdt9w" (UID: "374b7dce-d3a8-4665-a0ad-96c74a838738") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:08.765450 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:08.765407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:08.765612 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.765556    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:08.765612 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.765581    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:08.765612 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.765590    2580 projected.go:196] Error preparing data for projected volume kube-api-access-7bvgz for pod openshift-network-diagnostics/network-check-target-xwlqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:08.765733 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.765643    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz podName:45839b82-7bb9-4f27-8efe-89affa6fc06a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:24.765630219 +0000 UTC m=+34.326657514 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bvgz" (UniqueName: "kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz") pod "network-check-target-xwlqv" (UID: "45839b82-7bb9-4f27-8efe-89affa6fc06a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:08.930385 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:08.930355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:08.930385 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:08.930380    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:08.930600 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:08.930355    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:08.930600 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.930471    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:08.930600 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.930572    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:08.930755 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:08.930651    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:10.930960 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:10.930697    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:10.931401 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:10.930786    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:10.931401 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:10.931062    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:10.931401 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:10.930811    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:10.931401 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:10.931162    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:10.931401 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:10.931210    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:12.930791 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:12.930542    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:12.931338 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:12.931080    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:12.931338 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:12.930538    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:12.931338 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:12.931221    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:12.931338 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:12.930541    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:12.931524 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:12.931503    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:13.066882 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.066854    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 20:52:13.067185 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.067164    2580 generic.go:356] "Generic (PLEG): container finished" podID="326bde36-e763-476f-92d8-e411d84cad01" containerID="c1bda282ef3e99cfee26f8ee560ddb54eeda60956133ea0c43c50507a6956584" exitCode=1
Sep 09 20:52:13.067245 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.067225    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerStarted","Data":"f714811f34ea3373963274844421154c28a7b0af40c14fa39310b3c4b698d495"}
Sep 09 20:52:13.067315 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.067288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerStarted","Data":"6c0e0a020757bba8791d30a7ac8153bf0a0b94ced1e79392f741f9ea0d17487e"}
Sep 09 20:52:13.067424 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.067336    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerStarted","Data":"641a2131fe7d4dce32e0bc6065272bc232539a15246e714bf7494a1c09dfb4d1"}
Sep 09 20:52:13.067424 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.067377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerStarted","Data":"f62ecf1814476b6b79b6963bb625f6c6b3ed7e45482ba4147ef270dec39cb5c3"}
Sep 09 20:52:13.067536 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.067423    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerDied","Data":"c1bda282ef3e99cfee26f8ee560ddb54eeda60956133ea0c43c50507a6956584"}
Sep 09 20:52:13.067536 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.067470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerStarted","Data":"60262570aef17cf62a8549b2c33e4032857c352faa33c2b65d4a2c0ff5c3ee2c"}
Sep 09 20:52:13.068673 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.068656    2580 generic.go:356] "Generic (PLEG): container finished" podID="f3b6ca73-058e-4a22-9cfa-dbbf72f9564b" containerID="03db2ccf39caf393e2d14092a82b18e6b1b206c23642a612be735260ba4baaf7" exitCode=0
Sep 09 20:52:13.068730 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.068715    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerDied","Data":"03db2ccf39caf393e2d14092a82b18e6b1b206c23642a612be735260ba4baaf7"}
Sep 09 20:52:13.070317 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.070292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wh7mr" event={"ID":"fe5b2e20-19a6-4742-8f06-c1f10f896bb4","Type":"ContainerStarted","Data":"9723e6c9067e426c3ee40fb398dcb5daa899803910bda3cafdfb58eb3d8d6c49"}
Sep 09 20:52:13.071793 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.071747    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7zwsv" event={"ID":"9b73b24e-5ef0-4299-90d8-b7932046333b","Type":"ContainerStarted","Data":"aaa8a34342b0f1a3830e0d71d46c461346842c58cbb8c8c7b9544c0df3db565c"}
Sep 09 20:52:13.073336 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.073273    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr" event={"ID":"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e","Type":"ContainerStarted","Data":"db3a32624956041fcf6593afa2c4afc9bc0a9d445df55d2d0514b14ab47220f5"}
Sep 09 20:52:13.074707 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.074688    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xxnbh" event={"ID":"4df75668-d74f-4dde-81dd-ecac0a82140a","Type":"ContainerStarted","Data":"ecd4e46731758436e59f9239422170c0ec5296d737a095f9cf92d62e68074243"}
Sep 09 20:52:13.076527 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.076501    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-284xd" event={"ID":"8054739c-ef5e-4bfd-83ab-797a962f6b1b","Type":"ContainerStarted","Data":"a469e8dd0c1c1c2bf6ce14739e1326fad516394862571fc8fe20db4dcd5d7ff0"}
Sep 09 20:52:13.077861 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.077824    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q2tpw" event={"ID":"596b15b6-2d26-46db-a293-495852388c4f","Type":"ContainerStarted","Data":"d8cf0221249291846223883d51c82d26d28aa885e5957b8c4369e4f1450793d5"}
Sep 09 20:52:13.089444 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.089410    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-40.ec2.internal" podStartSLOduration=21.089380757 podStartE2EDuration="21.089380757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:56.047506794 +0000 UTC m=+5.608534116" watchObservedRunningTime="2026-09-09 20:52:13.089380757 +0000 UTC m=+22.650408055"
Sep 09 20:52:13.105793 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.105735    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-284xd" podStartSLOduration=4.155381597 podStartE2EDuration="22.105723767s" totalImagesPullingTime="17.95034217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:13.105355628 +0000 UTC m=+22.666382944" watchObservedRunningTime="2026-09-09 20:52:13.105723767 +0000 UTC m=+22.666751084"
Sep 09 20:52:13.120504 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.120467    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q2tpw" podStartSLOduration=4.552663358 podStartE2EDuration="22.120458012s" totalImagesPullingTime="17.567794654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:13.120024006 +0000 UTC m=+22.681051320" watchObservedRunningTime="2026-09-09 20:52:13.120458012 +0000 UTC m=+22.681485331"
Sep 09 20:52:13.135127 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.135093    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7zwsv" podStartSLOduration=3.5516458440000003 podStartE2EDuration="21.135085631s" totalImagesPullingTime="17.583439787s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:13.134417267 +0000 UTC m=+22.695444583" watchObservedRunningTime="2026-09-09 20:52:13.135085631 +0000 UTC m=+22.696112983"
Sep 09 20:52:13.160221 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.160185    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xxnbh" podStartSLOduration=4.5817872269999995 podStartE2EDuration="22.160172795s" totalImagesPullingTime="17.578385568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:13.159562106 +0000 UTC m=+22.720589422" watchObservedRunningTime="2026-09-09 20:52:13.160172795 +0000 UTC m=+22.721200112"
Sep 09 20:52:13.179244 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:13.179213    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wh7mr" podStartSLOduration=4.191820503 podStartE2EDuration="22.179204122s" totalImagesPullingTime="17.987383619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:13.17874186 +0000 UTC m=+22.739769177" watchObservedRunningTime="2026-09-09 20:52:13.179204122 +0000 UTC m=+22.740231438"
Sep 09 20:52:14.082624 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:14.082539    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ngnpn" event={"ID":"334cd7af-26a0-4b78-8683-bdfae8ae703e","Type":"ContainerStarted","Data":"d2e7b7a8187b9610af41a1f3654c493c04a6f31dc38decc052cb8c4826fb4643"}
Sep 09 20:52:14.097500 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:14.097461    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ngnpn" podStartSLOduration=5.151077573 podStartE2EDuration="23.097444348s" totalImagesPullingTime="17.946366775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:14.097215557 +0000 UTC m=+23.658242874" watchObservedRunningTime="2026-09-09 20:52:14.097444348 +0000 UTC m=+23.658471680"
Sep 09 20:52:14.925066 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:14.925036    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:52:14.932133 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:14.930534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:14.932133 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:14.930550    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:14.932133 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:14.930650    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:14.932133 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:14.930739    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:14.932133 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:14.930825    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:14.932133 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:14.930910    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:15.088349 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:15.088103    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 20:52:15.088796 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:15.088743    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerStarted","Data":"67000a6b7debaf5001024104d439dd1e85ca8437cbca6c0292786d391f0b5a1c"}
Sep 09 20:52:15.090950 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:15.090920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr" event={"ID":"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e","Type":"ContainerStarted","Data":"96e81f7f2b675146e8c1bfe9e4e917d8cc948f606d0dcf3e9045e97e14f1eaae"}
Sep 09 20:52:15.888892 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:15.888783    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:52:14.925063548Z","UUID":"adc277fb-dbf7-4bd7-8780-e301f472df49","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:52:15.891860 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:15.891609    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:52:15.891860 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:15.891642    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:52:16.744264 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:16.744233    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:52:16.745038 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:16.745014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:52:16.930445 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:16.930416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:16.930445 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:16.930451    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:16.930662 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:16.930417    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:16.930662 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:16.930546    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:16.930662 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:16.930632    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:16.930850 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:16.930700    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:17.099499 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:17.099451    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr" event={"ID":"7ab5d0cb-4e01-4fb5-96ff-acdd834c6d6e","Type":"ContainerStarted","Data":"9f8b79c5155f176ac6d6fbcb57fb040775cf6d0dbde158f7fcdf62d5736b9a65"}
Sep 09 20:52:17.099661 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:17.099616    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:52:17.100238 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:17.100217    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q2tpw"
Sep 09 20:52:17.132389 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:17.132338    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7tvgr" podStartSLOduration=4.74220146 podStartE2EDuration="26.132321831s" totalImagesPullingTime="21.390120371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:17.117098649 +0000 UTC m=+26.678125968" watchObservedRunningTime="2026-09-09 20:52:17.132321831 +0000 UTC m=+26.693349148"
Sep 09 20:52:18.107830 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:18.107665    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 20:52:18.108250 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:18.108222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerStarted","Data":"74e446d06002bed8b480a94347fc7c940aec9fc791810cc649dee22400f13023"}
Sep 09 20:52:18.108480 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:18.108460    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:52:18.108718 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:18.108705    2580 scope.go:117] "RemoveContainer" containerID="c1bda282ef3e99cfee26f8ee560ddb54eeda60956133ea0c43c50507a6956584"
Sep 09 20:52:18.111682 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:18.111642    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerStarted","Data":"2cd5adf02046f195336c8af21629939b4e8918a0ce715dc6faa21c7a66990c31"}
Sep 09 20:52:18.125009 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:18.124990    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:52:18.930289 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:18.930261    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:18.930484 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:18.930265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:18.930484 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:18.930363    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:18.930484 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:18.930266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:18.930484 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:18.930458    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:18.930625 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:18.930519    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:19.119508 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.119479    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 20:52:19.120060 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.120023    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" event={"ID":"326bde36-e763-476f-92d8-e411d84cad01","Type":"ContainerStarted","Data":"5c807b0953f2381f3fa70a7c633fb3ba192413c98cb15a52cf65bff143f1f992"}
Sep 09 20:52:19.120277 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.120259    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:52:19.120382 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.120370    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:52:19.122181 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.122155    2580 generic.go:356] "Generic (PLEG): container finished" podID="f3b6ca73-058e-4a22-9cfa-dbbf72f9564b" containerID="2cd5adf02046f195336c8af21629939b4e8918a0ce715dc6faa21c7a66990c31" exitCode=0
Sep 09 20:52:19.122330 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.122299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerDied","Data":"2cd5adf02046f195336c8af21629939b4e8918a0ce715dc6faa21c7a66990c31"}
Sep 09 20:52:19.138382 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.138350    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:52:19.155091 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.155029    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb" podStartSLOduration=10.208059274 podStartE2EDuration="28.155008523s" totalImagesPullingTime="17.946949249s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:19.15419699 +0000 UTC m=+28.715224320" watchObservedRunningTime="2026-09-09 20:52:19.155008523 +0000 UTC m=+28.716035842"
Sep 09 20:52:19.813211 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.813177    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nzvfm"]
Sep 09 20:52:19.813358 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.813306    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:19.813417 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:19.813398    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:19.816785 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.816737    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zdt9w"]
Sep 09 20:52:19.816937 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.816889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:19.817022 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:19.817001    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:19.817402 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.817379    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xwlqv"]
Sep 09 20:52:19.817495 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:19.817480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:19.817591 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:19.817572    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:20.931161 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:20.931122    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:20.931844 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:20.931250    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:21.128687 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:21.128656    2580 generic.go:356] "Generic (PLEG): container finished" podID="f3b6ca73-058e-4a22-9cfa-dbbf72f9564b" containerID="5979cbf8ae566490eabe8310ce157a4d3274341510b18a6ad6fb5ca1a197a984" exitCode=0
Sep 09 20:52:21.128830 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:21.128739    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerDied","Data":"5979cbf8ae566490eabe8310ce157a4d3274341510b18a6ad6fb5ca1a197a984"}
Sep 09 20:52:21.929602 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:21.929437    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:21.929692 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:21.929486    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:21.929753 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:21.929726    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:21.929858 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:21.929780    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:22.133053 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:22.133017    2580 generic.go:356] "Generic (PLEG): container finished" podID="f3b6ca73-058e-4a22-9cfa-dbbf72f9564b" containerID="e7ae478849418757d7dd32f06ba8a789ca19bb4accfc69503b1aeaa4314a49d7" exitCode=0
Sep 09 20:52:22.133550 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:22.133062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerDied","Data":"e7ae478849418757d7dd32f06ba8a789ca19bb4accfc69503b1aeaa4314a49d7"}
Sep 09 20:52:22.930152 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:22.930117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:22.930330 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:22.930237    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:23.930307 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:23.930268    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:23.930801 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:23.930270    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:23.930801 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:23.930416    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zdt9w" podUID="374b7dce-d3a8-4665-a0ad-96c74a838738"
Sep 09 20:52:23.930801 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:23.930519    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-nzvfm" podUID="b7b1474e-02e4-4041-a48a-2a3a652388c9"
Sep 09 20:52:24.572591 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:24.572555    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:24.572815 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:24.572679    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:24.572815 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:24.572739    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs podName:b7b1474e-02e4-4041-a48a-2a3a652388c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:56.572722402 +0000 UTC m=+66.133749701 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs") pod "network-metrics-daemon-nzvfm" (UID: "b7b1474e-02e4-4041-a48a-2a3a652388c9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:52:24.673048 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:24.673016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:24.673226 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:24.673129    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:24.673226 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:24.673198    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret podName:374b7dce-d3a8-4665-a0ad-96c74a838738 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:56.67317947 +0000 UTC m=+66.234206765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret") pod "global-pull-secret-syncer-zdt9w" (UID: "374b7dce-d3a8-4665-a0ad-96c74a838738") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:52:24.773809 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:24.773777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:24.773981 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:24.773891    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:52:24.773981 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:24.773920    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:52:24.773981 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:24.773934    2580 projected.go:196] Error preparing data for projected volume kube-api-access-7bvgz for pod openshift-network-diagnostics/network-check-target-xwlqv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:24.774113 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:24.774000    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz podName:45839b82-7bb9-4f27-8efe-89affa6fc06a nodeName:}" failed. No retries permitted until 2026-09-09 20:52:56.773980421 +0000 UTC m=+66.335007729 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7bvgz" (UniqueName: "kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz") pod "network-check-target-xwlqv" (UID: "45839b82-7bb9-4f27-8efe-89affa6fc06a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:52:24.930152 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:24.930117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:24.930305 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:24.930252    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xwlqv" podUID="45839b82-7bb9-4f27-8efe-89affa6fc06a"
Sep 09 20:52:25.254676 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.254606    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-40.ec2.internal" event="NodeReady"
Sep 09 20:52:25.255146 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.254773    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:52:25.291777 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.291736    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d4ddc64d9-gs6xr"]
Sep 09 20:52:25.314291 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.314266    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.316642 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.316068    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fffxj"]
Sep 09 20:52:25.328566 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.328543    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:52:25.328683 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.328614    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:52:25.328820 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.328738    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:52:25.328918 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.328876    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tn7rg\""
Sep 09 20:52:25.332920 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.332902    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:52:25.336116 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.336096    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fpc6w"]
Sep 09 20:52:25.336253 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.336236    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fffxj"
Sep 09 20:52:25.338576 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.338560    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:52:25.338678 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.338656    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:52:25.338830 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.338813    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:52:25.339009 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.338993    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gtzr8\""
Sep 09 20:52:25.353054 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.353033    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d4ddc64d9-gs6xr"]
Sep 09 20:52:25.353165 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.353120    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.357152 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.357132    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:52:25.357262 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.357213    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:52:25.357388 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.357373    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktpgk\""
Sep 09 20:52:25.359086 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.359065    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fpc6w"]
Sep 09 20:52:25.359182 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.359115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fffxj"]
Sep 09 20:52:25.378904 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.378884    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jxc\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-kube-api-access-l6jxc\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.379011 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.378925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-trusted-ca\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.379011 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.378971    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb70c45-bc88-4d76-8332-9c53b3a12c72-ca-trust-extracted\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.379011 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.378997    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-bound-sa-token\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.379145 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.379080    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.379145 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.379123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-installation-pull-secrets\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.379241 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.379146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-certificates\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.379241 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.379176    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-image-registry-private-configuration\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.480470 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480423    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.480659 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480501    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jxc\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-kube-api-access-l6jxc\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.480659 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtglj\" (UniqueName: \"kubernetes.io/projected/b817371a-9b4f-45c4-a96b-e4d22d616ca4-kube-api-access-vtglj\") pod \"ingress-canary-fffxj\" (UID: \"b817371a-9b4f-45c4-a96b-e4d22d616ca4\") " pod="openshift-ingress-canary/ingress-canary-fffxj"
Sep 09 20:52:25.480659 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-trusted-ca\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.480659 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb70c45-bc88-4d76-8332-9c53b3a12c72-ca-trust-extracted\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.480659 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-bound-sa-token\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.480659 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.481009 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480828    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-config-volume\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.481009 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-tmp-dir\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.481009 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480891    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-installation-pull-secrets\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.481009 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b817371a-9b4f-45c4-a96b-e4d22d616ca4-cert\") pod \"ingress-canary-fffxj\" (UID: \"b817371a-9b4f-45c4-a96b-e4d22d616ca4\") " pod="openshift-ingress-canary/ingress-canary-fffxj"
Sep 09 20:52:25.481009 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.480925    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbg86\" (UniqueName: \"kubernetes.io/projected/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-kube-api-access-sbg86\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.481250 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.481055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-certificates\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.481250 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.481100    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-image-registry-private-configuration\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.481250 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:25.481112    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:25.481250 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:25.481127    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4ddc64d9-gs6xr: secret "image-registry-tls" not found
Sep 09 20:52:25.481250 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:25.481177    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls podName:ecb70c45-bc88-4d76-8332-9c53b3a12c72 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:25.981160421 +0000 UTC m=+35.542187716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls") pod "image-registry-d4ddc64d9-gs6xr" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72") : secret "image-registry-tls" not found
Sep 09 20:52:25.481510 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.481060    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb70c45-bc88-4d76-8332-9c53b3a12c72-ca-trust-extracted\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.481686 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.481667    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-certificates\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.481784 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.481670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-trusted-ca\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.485738 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.485714    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-installation-pull-secrets\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.486415 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.486390    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-image-registry-private-configuration\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.492321 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.492301    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-bound-sa-token\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.510017 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.509961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jxc\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-kube-api-access-l6jxc\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.581916 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.581872    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-config-volume\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.582103 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.581928    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-tmp-dir\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.582103 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.581967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b817371a-9b4f-45c4-a96b-e4d22d616ca4-cert\") pod \"ingress-canary-fffxj\" (UID: \"b817371a-9b4f-45c4-a96b-e4d22d616ca4\") " pod="openshift-ingress-canary/ingress-canary-fffxj"
Sep 09 20:52:25.582103 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.581987    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbg86\" (UniqueName: \"kubernetes.io/projected/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-kube-api-access-sbg86\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.582103 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.582041    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.582103 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.582066    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtglj\" (UniqueName: \"kubernetes.io/projected/b817371a-9b4f-45c4-a96b-e4d22d616ca4-kube-api-access-vtglj\") pod \"ingress-canary-fffxj\" (UID: \"b817371a-9b4f-45c4-a96b-e4d22d616ca4\") " pod="openshift-ingress-canary/ingress-canary-fffxj"
Sep 09 20:52:25.582293 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:25.582283    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:25.582334 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.582307    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-tmp-dir\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.582382 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:25.582352    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls podName:e815f55d-a4cf-463f-8190-b8fe0f3bd98c nodeName:}" failed. No retries permitted until 2026-09-09 20:52:26.082321435 +0000 UTC m=+35.643348731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls") pod "dns-default-fpc6w" (UID: "e815f55d-a4cf-463f-8190-b8fe0f3bd98c") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:25.582574 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.582546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-config-volume\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.584878 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.584852    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b817371a-9b4f-45c4-a96b-e4d22d616ca4-cert\") pod \"ingress-canary-fffxj\" (UID: \"b817371a-9b4f-45c4-a96b-e4d22d616ca4\") " pod="openshift-ingress-canary/ingress-canary-fffxj"
Sep 09 20:52:25.592684 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.592660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtglj\" (UniqueName: \"kubernetes.io/projected/b817371a-9b4f-45c4-a96b-e4d22d616ca4-kube-api-access-vtglj\") pod \"ingress-canary-fffxj\" (UID: \"b817371a-9b4f-45c4-a96b-e4d22d616ca4\") " pod="openshift-ingress-canary/ingress-canary-fffxj"
Sep 09 20:52:25.594133 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.594109    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbg86\" (UniqueName: \"kubernetes.io/projected/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-kube-api-access-sbg86\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:25.648704 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.648667    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fffxj"
Sep 09 20:52:25.929795 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.929750    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:25.930000 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.929750    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:25.932381 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.932357    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:52:25.932501 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.932479    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:52:25.932593 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.932574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2kpj\""
Sep 09 20:52:25.984742 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:25.984705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:25.984941 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:25.984924    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:25.984941 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:25.984942    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4ddc64d9-gs6xr: secret "image-registry-tls" not found
Sep 09 20:52:25.985050 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:25.985011    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls podName:ecb70c45-bc88-4d76-8332-9c53b3a12c72 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:26.984990788 +0000 UTC m=+36.546018092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls") pod "image-registry-d4ddc64d9-gs6xr" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72") : secret "image-registry-tls" not found
Sep 09 20:52:26.085672 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:26.085637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:26.085850 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:26.085826    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:26.085946 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:26.085917    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls podName:e815f55d-a4cf-463f-8190-b8fe0f3bd98c nodeName:}" failed. No retries permitted until 2026-09-09 20:52:27.0858937 +0000 UTC m=+36.646920997 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls") pod "dns-default-fpc6w" (UID: "e815f55d-a4cf-463f-8190-b8fe0f3bd98c") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:26.845055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:26.844989    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fffxj"]
Sep 09 20:52:26.856695 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:52:26.856662    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb817371a_9b4f_45c4_a96b_e4d22d616ca4.slice/crio-3458da18b21c538995f148c73026f522fdf4142c3505bcf1d6f6d9a479533a8b WatchSource:0}: Error finding container 3458da18b21c538995f148c73026f522fdf4142c3505bcf1d6f6d9a479533a8b: Status 404 returned error can't find the container with id 3458da18b21c538995f148c73026f522fdf4142c3505bcf1d6f6d9a479533a8b
Sep 09 20:52:26.930577 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:26.930551    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:26.933093 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:26.933069    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:52:26.933167 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:26.933125    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:52:26.933787 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:26.933749    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74tsk\""
Sep 09 20:52:26.993718 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:26.993556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:26.993833 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:26.993697    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:26.993833 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:26.993781    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4ddc64d9-gs6xr: secret "image-registry-tls" not found
Sep 09 20:52:26.993833 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:26.993832    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls podName:ecb70c45-bc88-4d76-8332-9c53b3a12c72 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:28.99381702 +0000 UTC m=+38.554844315 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls") pod "image-registry-d4ddc64d9-gs6xr" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72") : secret "image-registry-tls" not found
Sep 09 20:52:27.095190 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:27.095113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:27.095330 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:27.095280    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:27.095390 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:27.095358    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls podName:e815f55d-a4cf-463f-8190-b8fe0f3bd98c nodeName:}" failed. No retries permitted until 2026-09-09 20:52:29.095336785 +0000 UTC m=+38.656364097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls") pod "dns-default-fpc6w" (UID: "e815f55d-a4cf-463f-8190-b8fe0f3bd98c") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:27.146800 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:27.146750    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fffxj" event={"ID":"b817371a-9b4f-45c4-a96b-e4d22d616ca4","Type":"ContainerStarted","Data":"3458da18b21c538995f148c73026f522fdf4142c3505bcf1d6f6d9a479533a8b"}
Sep 09 20:52:27.149882 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:27.149849    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerStarted","Data":"aec44e0a9106196e069adcc70c5c9007f85d9acb7f08c88c890c527b701414ca"}
Sep 09 20:52:28.157837 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:28.157802    2580 generic.go:356] "Generic (PLEG): container finished" podID="f3b6ca73-058e-4a22-9cfa-dbbf72f9564b" containerID="aec44e0a9106196e069adcc70c5c9007f85d9acb7f08c88c890c527b701414ca" exitCode=0
Sep 09 20:52:28.158313 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:28.157856    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerDied","Data":"aec44e0a9106196e069adcc70c5c9007f85d9acb7f08c88c890c527b701414ca"}
Sep 09 20:52:29.011179 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:29.011144    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:29.011374 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:29.011327    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:29.011374 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:29.011350    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4ddc64d9-gs6xr: secret "image-registry-tls" not found
Sep 09 20:52:29.011490 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:29.011426    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls podName:ecb70c45-bc88-4d76-8332-9c53b3a12c72 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:33.01140468 +0000 UTC m=+42.572431976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls") pod "image-registry-d4ddc64d9-gs6xr" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72") : secret "image-registry-tls" not found
Sep 09 20:52:29.112441 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:29.112400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:29.112609 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:29.112554    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:29.112662 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:29.112626    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls podName:e815f55d-a4cf-463f-8190-b8fe0f3bd98c nodeName:}" failed. No retries permitted until 2026-09-09 20:52:33.112610152 +0000 UTC m=+42.673637448 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls") pod "dns-default-fpc6w" (UID: "e815f55d-a4cf-463f-8190-b8fe0f3bd98c") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:29.164823 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:29.164790    2580 generic.go:356] "Generic (PLEG): container finished" podID="f3b6ca73-058e-4a22-9cfa-dbbf72f9564b" containerID="7ca82437ef24149a8618cc978376acd30abe4b63c6876e1bccef920f877657c8" exitCode=0
Sep 09 20:52:29.165377 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:29.164855    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerDied","Data":"7ca82437ef24149a8618cc978376acd30abe4b63c6876e1bccef920f877657c8"}
Sep 09 20:52:30.172023 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:30.171929    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6phgb" event={"ID":"f3b6ca73-058e-4a22-9cfa-dbbf72f9564b","Type":"ContainerStarted","Data":"c53c38d7d444f78f92c566b3c5d6768ca4b48a5d8781f77a6a2400da0ac0fcf5"}
Sep 09 20:52:30.173457 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:30.173436    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fffxj" event={"ID":"b817371a-9b4f-45c4-a96b-e4d22d616ca4","Type":"ContainerStarted","Data":"2a592b68c8197213043bddfa2a15cf71efb22d5c7ef21354f8a391530b755e81"}
Sep 09 20:52:30.192400 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:30.192361    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6phgb" podStartSLOduration=9.159992711 podStartE2EDuration="39.192350046s" totalImagesPullingTime="30.032357335s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:30.191174608 +0000 UTC m=+39.752201927" watchObservedRunningTime="2026-09-09 20:52:30.192350046 +0000 UTC m=+39.753377362"
Sep 09 20:52:30.206820 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:30.206777    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fffxj" podStartSLOduration=2.199860227 podStartE2EDuration="5.206738771s" totalImagesPullingTime="3.006878544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:30.20614493 +0000 UTC m=+39.767172246" watchObservedRunningTime="2026-09-09 20:52:30.206738771 +0000 UTC m=+39.767766086"
Sep 09 20:52:33.044408 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:33.044371    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:33.044823 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:33.044519    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:33.044823 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:33.044535    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4ddc64d9-gs6xr: secret "image-registry-tls" not found
Sep 09 20:52:33.044823 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:33.044596    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls podName:ecb70c45-bc88-4d76-8332-9c53b3a12c72 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:41.044582432 +0000 UTC m=+50.605609727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls") pod "image-registry-d4ddc64d9-gs6xr" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72") : secret "image-registry-tls" not found
Sep 09 20:52:33.145324 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:33.145292    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:33.145474 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:33.145403    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:33.145474 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:33.145453    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls podName:e815f55d-a4cf-463f-8190-b8fe0f3bd98c nodeName:}" failed. No retries permitted until 2026-09-09 20:52:41.145439504 +0000 UTC m=+50.706466803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls") pod "dns-default-fpc6w" (UID: "e815f55d-a4cf-463f-8190-b8fe0f3bd98c") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:41.098504 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:41.098463    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:41.099013 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:41.098576    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:41.099013 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:41.098588    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4ddc64d9-gs6xr: secret "image-registry-tls" not found
Sep 09 20:52:41.099013 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:41.098648    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls podName:ecb70c45-bc88-4d76-8332-9c53b3a12c72 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:57.098633833 +0000 UTC m=+66.659661127 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls") pod "image-registry-d4ddc64d9-gs6xr" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72") : secret "image-registry-tls" not found
Sep 09 20:52:41.198976 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:41.198945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:41.199127 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:41.199082    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:41.199166 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:41.199138    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls podName:e815f55d-a4cf-463f-8190-b8fe0f3bd98c nodeName:}" failed. No retries permitted until 2026-09-09 20:52:57.199123221 +0000 UTC m=+66.760150516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls") pod "dns-default-fpc6w" (UID: "e815f55d-a4cf-463f-8190-b8fe0f3bd98c") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:44.724278 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.724242    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"]
Sep 09 20:52:44.740218 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.740192    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h"]
Sep 09 20:52:44.740412 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.740324    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:44.743236 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.742796    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:52:44.743236 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.742944    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-r2f8m\""
Sep 09 20:52:44.743236 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.743145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 20:52:44.744491 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.744474    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:52:44.744595 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.744559    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 20:52:44.752437 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.752416    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"]
Sep 09 20:52:44.752534 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.752449    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h"]
Sep 09 20:52:44.752534 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.752521    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h"
Sep 09 20:52:44.754530 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.754513    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:44.754684 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.754604    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:44.754939 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.754917    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-b2fsn\""
Sep 09 20:52:44.924541 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.924507    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:44.924541 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.924542    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnff8\" (UniqueName: \"kubernetes.io/projected/d834aab5-aae4-4735-a1a0-a375d3f37d09-kube-api-access-hnff8\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:44.924779 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.924578    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d834aab5-aae4-4735-a1a0-a375d3f37d09-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:44.924779 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:44.924616    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwfhl\" (UniqueName: \"kubernetes.io/projected/814516c5-5564-4b9b-9753-883546d6093e-kube-api-access-zwfhl\") pod \"volume-data-source-validator-695c895bd6-7wc7h\" (UID: \"814516c5-5564-4b9b-9753-883546d6093e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h"
Sep 09 20:52:45.025955 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.025870    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d834aab5-aae4-4735-a1a0-a375d3f37d09-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:45.025955 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.025919    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwfhl\" (UniqueName: \"kubernetes.io/projected/814516c5-5564-4b9b-9753-883546d6093e-kube-api-access-zwfhl\") pod \"volume-data-source-validator-695c895bd6-7wc7h\" (UID: \"814516c5-5564-4b9b-9753-883546d6093e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h"
Sep 09 20:52:45.026189 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.026000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:45.026189 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.026022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnff8\" (UniqueName: \"kubernetes.io/projected/d834aab5-aae4-4735-a1a0-a375d3f37d09-kube-api-access-hnff8\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:45.026189 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:45.026122    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:45.026334 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:45.026210    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls podName:d834aab5-aae4-4735-a1a0-a375d3f37d09 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:45.526184823 +0000 UTC m=+55.087212142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-z66wv" (UID: "d834aab5-aae4-4735-a1a0-a375d3f37d09") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:45.026697 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.026677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d834aab5-aae4-4735-a1a0-a375d3f37d09-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:45.035255 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.035228    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwfhl\" (UniqueName: \"kubernetes.io/projected/814516c5-5564-4b9b-9753-883546d6093e-kube-api-access-zwfhl\") pod \"volume-data-source-validator-695c895bd6-7wc7h\" (UID: \"814516c5-5564-4b9b-9753-883546d6093e\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h"
Sep 09 20:52:45.041035 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.041017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnff8\" (UniqueName: \"kubernetes.io/projected/d834aab5-aae4-4735-a1a0-a375d3f37d09-kube-api-access-hnff8\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:45.061306 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.061286    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h"
Sep 09 20:52:45.342026 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.341979    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h"]
Sep 09 20:52:45.346639 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:52:45.346604    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod814516c5_5564_4b9b_9753_883546d6093e.slice/crio-e0b05339af8eba4004508f75ffdeb7b72641ffd5b28f09e24030f569ee6f5fda WatchSource:0}: Error finding container e0b05339af8eba4004508f75ffdeb7b72641ffd5b28f09e24030f569ee6f5fda: Status 404 returned error can't find the container with id e0b05339af8eba4004508f75ffdeb7b72641ffd5b28f09e24030f569ee6f5fda
Sep 09 20:52:45.531154 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:45.531110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:45.531354 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:45.531301    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:45.531420 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:45.531400    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls podName:d834aab5-aae4-4735-a1a0-a375d3f37d09 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:46.531376813 +0000 UTC m=+56.092404123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-z66wv" (UID: "d834aab5-aae4-4735-a1a0-a375d3f37d09") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:46.211640 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:46.211607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h" event={"ID":"814516c5-5564-4b9b-9753-883546d6093e","Type":"ContainerStarted","Data":"e0b05339af8eba4004508f75ffdeb7b72641ffd5b28f09e24030f569ee6f5fda"}
Sep 09 20:52:46.538673 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:46.538650    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:46.538803 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:46.538790    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:46.538864 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:46.538855    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls podName:d834aab5-aae4-4735-a1a0-a375d3f37d09 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:48.538835135 +0000 UTC m=+58.099862438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-z66wv" (UID: "d834aab5-aae4-4735-a1a0-a375d3f37d09") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:47.215969 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:47.215935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h" event={"ID":"814516c5-5564-4b9b-9753-883546d6093e","Type":"ContainerStarted","Data":"81e7b03040c43f181ca7fbb8c0b9a769d475f174bf6da73284addd5ed1be524c"}
Sep 09 20:52:47.233427 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:47.233381    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-695c895bd6-7wc7h" podStartSLOduration=2.103788446 podStartE2EDuration="3.233364977s" totalImagesPullingTime="1.129576531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:47.233232179 +0000 UTC m=+56.794259496" watchObservedRunningTime="2026-09-09 20:52:47.233364977 +0000 UTC m=+56.794392296"
Sep 09 20:52:48.553867 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:48.553831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:48.554248 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:48.553933    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:48.554248 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:48.553986    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls podName:d834aab5-aae4-4735-a1a0-a375d3f37d09 nodeName:}" failed. No retries permitted until 2026-09-09 20:52:52.553973168 +0000 UTC m=+62.115000463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-z66wv" (UID: "d834aab5-aae4-4735-a1a0-a375d3f37d09") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:49.614176 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.614143    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"]
Sep 09 20:52:49.616106 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.616090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.619737 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.619714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:52:49.619884 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.619753    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 09 20:52:49.619884 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.619777    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 09 20:52:49.619884 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.619854    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-b9cdl\""
Sep 09 20:52:49.620321 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.620305    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 09 20:52:49.632688 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.632663    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"]
Sep 09 20:52:49.647750 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.647713    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bthvc"]
Sep 09 20:52:49.649925 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.649908    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.651788 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.651751    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:52:49.651884 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.651771    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:52:49.651980 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.651965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:52:49.652124 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.652111    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5vkbt\""
Sep 09 20:52:49.652174 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.652138    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:52:49.665906 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.665882    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bthvc"]
Sep 09 20:52:49.763794 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.763740    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-crio-socket\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.763978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.763818    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.763978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.763859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldllq\" (UniqueName: \"kubernetes.io/projected/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-api-access-ldllq\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.763978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.763879    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04dbd6bf-11a8-4786-83d2-6e8df4706cfd-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mnx76\" (UID: \"04dbd6bf-11a8-4786-83d2-6e8df4706cfd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.763978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.763893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-data-volume\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.763978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.763957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04dbd6bf-11a8-4786-83d2-6e8df4706cfd-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mnx76\" (UID: \"04dbd6bf-11a8-4786-83d2-6e8df4706cfd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.764147 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.763987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzf7p\" (UniqueName: \"kubernetes.io/projected/04dbd6bf-11a8-4786-83d2-6e8df4706cfd-kube-api-access-vzf7p\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mnx76\" (UID: \"04dbd6bf-11a8-4786-83d2-6e8df4706cfd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.764147 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.764044    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.865381 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865304    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.865381 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldllq\" (UniqueName: \"kubernetes.io/projected/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-api-access-ldllq\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.865515 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04dbd6bf-11a8-4786-83d2-6e8df4706cfd-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mnx76\" (UID: \"04dbd6bf-11a8-4786-83d2-6e8df4706cfd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.865515 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-data-volume\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.865515 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:49.865445    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:49.865515 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04dbd6bf-11a8-4786-83d2-6e8df4706cfd-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mnx76\" (UID: \"04dbd6bf-11a8-4786-83d2-6e8df4706cfd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.865676 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865516    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzf7p\" (UniqueName: \"kubernetes.io/projected/04dbd6bf-11a8-4786-83d2-6e8df4706cfd-kube-api-access-vzf7p\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mnx76\" (UID: \"04dbd6bf-11a8-4786-83d2-6e8df4706cfd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.865676 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:49.865532    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:50.365512503 +0000 UTC m=+59.926539824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:49.865676 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.865676 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-crio-socket\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.865853 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-crio-socket\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.865853 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:49.865735    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:49.865911 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:49.865858    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:50.36583794 +0000 UTC m=+59.926865237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:49.865974 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.865956    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-data-volume\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.866059 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.866043    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/04dbd6bf-11a8-4786-83d2-6e8df4706cfd-config\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mnx76\" (UID: \"04dbd6bf-11a8-4786-83d2-6e8df4706cfd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.867924 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.867897    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/04dbd6bf-11a8-4786-83d2-6e8df4706cfd-serving-cert\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mnx76\" (UID: \"04dbd6bf-11a8-4786-83d2-6e8df4706cfd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.874990 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.874970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzf7p\" (UniqueName: \"kubernetes.io/projected/04dbd6bf-11a8-4786-83d2-6e8df4706cfd-kube-api-access-vzf7p\") pod \"kube-storage-version-migrator-operator-8d7dd4675-mnx76\" (UID: \"04dbd6bf-11a8-4786-83d2-6e8df4706cfd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.874990 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.874986    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldllq\" (UniqueName: \"kubernetes.io/projected/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-api-access-ldllq\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:49.924086 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.924058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"
Sep 09 20:52:49.988404 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.988370    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-qd7jf"]
Sep 09 20:52:49.990925 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.990901    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-qd7jf"
Sep 09 20:52:49.993102 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:49.993080    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rs8fq\""
Sep 09 20:52:50.000503 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.000483    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-qd7jf"]
Sep 09 20:52:50.168720 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.168643    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9gh\" (UniqueName: \"kubernetes.io/projected/fcc45c41-2627-41ac-97fa-7baaa2cce355-kube-api-access-zq9gh\") pod \"network-check-source-685456655f-qd7jf\" (UID: \"fcc45c41-2627-41ac-97fa-7baaa2cce355\") " pod="openshift-network-diagnostics/network-check-source-685456655f-qd7jf"
Sep 09 20:52:50.213516 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.213481    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76"]
Sep 09 20:52:50.217058 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:52:50.217034    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dbd6bf_11a8_4786_83d2_6e8df4706cfd.slice/crio-f37ca7ac7f248ea34d1e6904146525b76f9bd052240e85d7fff7d864a1c6b885 WatchSource:0}: Error finding container f37ca7ac7f248ea34d1e6904146525b76f9bd052240e85d7fff7d864a1c6b885: Status 404 returned error can't find the container with id f37ca7ac7f248ea34d1e6904146525b76f9bd052240e85d7fff7d864a1c6b885
Sep 09 20:52:50.224260 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.224237    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76" event={"ID":"04dbd6bf-11a8-4786-83d2-6e8df4706cfd","Type":"ContainerStarted","Data":"f37ca7ac7f248ea34d1e6904146525b76f9bd052240e85d7fff7d864a1c6b885"}
Sep 09 20:52:50.269657 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.269620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9gh\" (UniqueName: \"kubernetes.io/projected/fcc45c41-2627-41ac-97fa-7baaa2cce355-kube-api-access-zq9gh\") pod \"network-check-source-685456655f-qd7jf\" (UID: \"fcc45c41-2627-41ac-97fa-7baaa2cce355\") " pod="openshift-network-diagnostics/network-check-source-685456655f-qd7jf"
Sep 09 20:52:50.279285 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.279255    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9gh\" (UniqueName: \"kubernetes.io/projected/fcc45c41-2627-41ac-97fa-7baaa2cce355-kube-api-access-zq9gh\") pod \"network-check-source-685456655f-qd7jf\" (UID: \"fcc45c41-2627-41ac-97fa-7baaa2cce355\") " pod="openshift-network-diagnostics/network-check-source-685456655f-qd7jf"
Sep 09 20:52:50.300221 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.300199    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-qd7jf"
Sep 09 20:52:50.370278 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.370239    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:50.370392 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.370332    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:50.370392 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:50.370383    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:50.370460 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:50.370426    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:50.370460 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:50.370455    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:51.370436803 +0000 UTC m=+60.931464104 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:50.370528 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:50.370473    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:51.370461927 +0000 UTC m=+60.931489268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:50.569201 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:50.569121    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-qd7jf"]
Sep 09 20:52:50.572784 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:52:50.572739    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc45c41_2627_41ac_97fa_7baaa2cce355.slice/crio-e32c2adebed7d749c22dab45cf833dbb237b8188e252d1c658d72af88b3be2d5 WatchSource:0}: Error finding container e32c2adebed7d749c22dab45cf833dbb237b8188e252d1c658d72af88b3be2d5: Status 404 returned error can't find the container with id e32c2adebed7d749c22dab45cf833dbb237b8188e252d1c658d72af88b3be2d5
Sep 09 20:52:51.142095 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:51.142064    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fc6xb"
Sep 09 20:52:51.229410 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:51.229351    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-qd7jf" event={"ID":"fcc45c41-2627-41ac-97fa-7baaa2cce355","Type":"ContainerStarted","Data":"e32c2adebed7d749c22dab45cf833dbb237b8188e252d1c658d72af88b3be2d5"}
Sep 09 20:52:51.377559 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:51.377523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:51.377730 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:51.377647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:51.377730 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:51.377663    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:51.377839 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:51.377737    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:53.377717517 +0000 UTC m=+62.938744818 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:51.377839 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:51.377754    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:51.377839 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:51.377836    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:53.377821339 +0000 UTC m=+62.938848660 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:51.384971 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:51.384946    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xxnbh_4df75668-d74f-4dde-81dd-ecac0a82140a/dns-node-resolver/0.log"
Sep 09 20:52:51.784987 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:51.784955    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7zwsv_9b73b24e-5ef0-4299-90d8-b7932046333b/node-ca/0.log"
Sep 09 20:52:52.584203 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:52.584122    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fffxj_b817371a-9b4f-45c4-a96b-e4d22d616ca4/serve-healthcheck-canary/0.log"
Sep 09 20:52:52.585293 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:52.585263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:52:52.585441 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:52.585418    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:52.585505 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:52.585488    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls podName:d834aab5-aae4-4735-a1a0-a375d3f37d09 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:00.585473685 +0000 UTC m=+70.146500982 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-z66wv" (UID: "d834aab5-aae4-4735-a1a0-a375d3f37d09") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:52:53.237264 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:53.237226    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76" event={"ID":"04dbd6bf-11a8-4786-83d2-6e8df4706cfd","Type":"ContainerStarted","Data":"ccad9ec14ed05ae84dad61121055bbe1e595ee13a7b081af48b28536e578b067"}
Sep 09 20:52:53.254191 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:53.254129    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76" podStartSLOduration=2.176391652 podStartE2EDuration="4.254109706s" totalImagesPullingTime="2.077718054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:53.252711085 +0000 UTC m=+62.813738428" watchObservedRunningTime="2026-09-09 20:52:53.254109706 +0000 UTC m=+62.815137022"
Sep 09 20:52:53.392456 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:53.392417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:53.392629 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:53.392495    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:53.392629 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:53.392571    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:53.392629 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:53.392571    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:53.392787 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:53.392640    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:57.392621891 +0000 UTC m=+66.953649200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:53.392787 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:53.392659    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:52:57.392650087 +0000 UTC m=+66.953677382 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:54.241870 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:54.241836    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-qd7jf" event={"ID":"fcc45c41-2627-41ac-97fa-7baaa2cce355","Type":"ContainerStarted","Data":"9d1b62bf928a032ccfc2fdefbea8789afadde1a84a11f660dde471f9f69ec34f"}
Sep 09 20:52:54.263490 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:54.263446    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-qd7jf" podStartSLOduration=1.716852984 podStartE2EDuration="5.263432508s" totalImagesPullingTime="3.546579524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:54.261940664 +0000 UTC m=+63.822967980" watchObservedRunningTime="2026-09-09 20:52:54.263432508 +0000 UTC m=+63.824459825"
Sep 09 20:52:56.617119 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.617082    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:52:56.620199 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.620182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:52:56.627989 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:56.627976    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:52:56.628040 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:56.628030    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs podName:b7b1474e-02e4-4041-a48a-2a3a652388c9 nodeName:}" failed. No retries permitted until 2026-09-09 20:54:00.628015509 +0000 UTC m=+130.189042808 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs") pod "network-metrics-daemon-nzvfm" (UID: "b7b1474e-02e4-4041-a48a-2a3a652388c9") : secret "metrics-daemon-secret" not found
Sep 09 20:52:56.717976 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.717943    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:56.720442 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.720427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:52:56.730625 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.730599    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/374b7dce-d3a8-4665-a0ad-96c74a838738-original-pull-secret\") pod \"global-pull-secret-syncer-zdt9w\" (UID: \"374b7dce-d3a8-4665-a0ad-96c74a838738\") " pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:56.818525 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.818493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:56.821102 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.821082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvgz\" (UniqueName: \"kubernetes.io/projected/45839b82-7bb9-4f27-8efe-89affa6fc06a-kube-api-access-7bvgz\") pod \"network-check-target-xwlqv\" (UID: \"45839b82-7bb9-4f27-8efe-89affa6fc06a\") " pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:56.840450 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.840429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zdt9w"
Sep 09 20:52:56.947628 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.947603    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74tsk\""
Sep 09 20:52:56.955464 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:56.955445    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:52:57.121506 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:57.121467    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:52:57.121671 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:57.121627    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:52:57.121671 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:57.121648    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d4ddc64d9-gs6xr: secret "image-registry-tls" not found
Sep 09 20:52:57.121741 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:57.121705    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls podName:ecb70c45-bc88-4d76-8332-9c53b3a12c72 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:29.121688942 +0000 UTC m=+98.682716240 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls") pod "image-registry-d4ddc64d9-gs6xr" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72") : secret "image-registry-tls" not found
Sep 09 20:52:57.144602 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:57.144533    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zdt9w"]
Sep 09 20:52:57.149448 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:52:57.149428    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374b7dce_d3a8_4665_a0ad_96c74a838738.slice/crio-117d7a508ef060cafb643f2bdefeb921b430f54a1b2b954d85ffdabdb5c811a1 WatchSource:0}: Error finding container 117d7a508ef060cafb643f2bdefeb921b430f54a1b2b954d85ffdabdb5c811a1: Status 404 returned error can't find the container with id 117d7a508ef060cafb643f2bdefeb921b430f54a1b2b954d85ffdabdb5c811a1
Sep 09 20:52:57.221889 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:57.221859    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:52:57.222045 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:57.222026    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:52:57.222102 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:57.222092    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls podName:e815f55d-a4cf-463f-8190-b8fe0f3bd98c nodeName:}" failed. No retries permitted until 2026-09-09 20:53:29.222075644 +0000 UTC m=+98.783102938 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls") pod "dns-default-fpc6w" (UID: "e815f55d-a4cf-463f-8190-b8fe0f3bd98c") : secret "dns-default-metrics-tls" not found
Sep 09 20:52:57.243392 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:57.243364    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xwlqv"]
Sep 09 20:52:57.246184 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:52:57.246163    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45839b82_7bb9_4f27_8efe_89affa6fc06a.slice/crio-d9c30113e71dd228884579c45d818abf5d764d7ee7ac5083827d5359ac7c64e4 WatchSource:0}: Error finding container d9c30113e71dd228884579c45d818abf5d764d7ee7ac5083827d5359ac7c64e4: Status 404 returned error can't find the container with id d9c30113e71dd228884579c45d818abf5d764d7ee7ac5083827d5359ac7c64e4
Sep 09 20:52:57.250085 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:57.250058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zdt9w" event={"ID":"374b7dce-d3a8-4665-a0ad-96c74a838738","Type":"ContainerStarted","Data":"117d7a508ef060cafb643f2bdefeb921b430f54a1b2b954d85ffdabdb5c811a1"}
Sep 09 20:52:57.251179 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:57.251162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xwlqv" event={"ID":"45839b82-7bb9-4f27-8efe-89affa6fc06a","Type":"ContainerStarted","Data":"d9c30113e71dd228884579c45d818abf5d764d7ee7ac5083827d5359ac7c64e4"}
Sep 09 20:52:57.423299 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:57.423207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:57.423299 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:57.423259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:52:57.423496 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:57.423341    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:52:57.423496 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:57.423374    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:57.423496 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:57.423396    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.423381151 +0000 UTC m=+74.984408446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : configmap "kube-rbac-proxy" not found
Sep 09 20:52:57.423496 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:52:57.423430    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:53:05.423416924 +0000 UTC m=+74.984444220 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:52:58.256361 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:58.256328    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xwlqv" event={"ID":"45839b82-7bb9-4f27-8efe-89affa6fc06a","Type":"ContainerStarted","Data":"5c2de78a826bb02f9364f74b41bf378c018c9e7cab594b6bb6e8b936f275782f"}
Sep 09 20:52:58.256714 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:52:58.256459    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:53:00.650203 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:00.650165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:53:00.650594 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:00.650289    2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:00.650594 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:00.650380    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls podName:d834aab5-aae4-4735-a1a0-a375d3f37d09 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:16.650358287 +0000 UTC m=+86.211385594 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-z66wv" (UID: "d834aab5-aae4-4735-a1a0-a375d3f37d09") : secret "cluster-monitoring-operator-tls" not found
Sep 09 20:53:02.274919 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:02.274885    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zdt9w" event={"ID":"374b7dce-d3a8-4665-a0ad-96c74a838738","Type":"ContainerStarted","Data":"90660d9e6c433d5c5d5a68f33e24ae3dc5edccca773e27ce7e5471cbac8d5edb"}
Sep 09 20:53:02.292515 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:02.292459    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zdt9w" podStartSLOduration=65.808092633 podStartE2EDuration="1m10.292442754s" totalImagesPullingTime="4.484350121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:02.291787919 +0000 UTC m=+71.852815236" watchObservedRunningTime="2026-09-09 20:53:02.292442754 +0000 UTC m=+71.853470072"
Sep 09 20:53:02.292712 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:02.292688    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xwlqv" podStartSLOduration=70.292680408 podStartE2EDuration="1m10.292680408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:58.277472013 +0000 UTC m=+67.838499329" watchObservedRunningTime="2026-09-09 20:53:02.292680408 +0000 UTC m=+71.853707736"
Sep 09 20:53:05.487325 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:05.487264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:53:05.487836 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:05.487362    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:53:05.487836 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:05.487433    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:05.487836 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:05.487498    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls podName:1b67f017-f68f-4ced-85c4-01baa0fc1d0f nodeName:}" failed. No retries permitted until 2026-09-09 20:53:21.487483953 +0000 UTC m=+91.048511252 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bthvc" (UID: "1b67f017-f68f-4ced-85c4-01baa0fc1d0f") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:53:05.488078 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:05.488053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:53:16.673380 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:16.673219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:53:16.675922 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:16.675896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d834aab5-aae4-4735-a1a0-a375d3f37d09-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-z66wv\" (UID: \"d834aab5-aae4-4735-a1a0-a375d3f37d09\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:53:16.853750 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:16.853713    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-r2f8m\""
Sep 09 20:53:16.861558 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:16.861537    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"
Sep 09 20:53:17.132913 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:17.132775    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv"]
Sep 09 20:53:17.135508 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:17.135472    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd834aab5_aae4_4735_a1a0_a375d3f37d09.slice/crio-36cbd5222994aee683a96cb4a4d58463fc7846fc0e071d62cd6770f729350dbc WatchSource:0}: Error finding container 36cbd5222994aee683a96cb4a4d58463fc7846fc0e071d62cd6770f729350dbc: Status 404 returned error can't find the container with id 36cbd5222994aee683a96cb4a4d58463fc7846fc0e071d62cd6770f729350dbc
Sep 09 20:53:17.319545 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:17.319516    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv" event={"ID":"d834aab5-aae4-4735-a1a0-a375d3f37d09","Type":"ContainerStarted","Data":"36cbd5222994aee683a96cb4a4d58463fc7846fc0e071d62cd6770f729350dbc"}
Sep 09 20:53:18.611930 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.611892    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"]
Sep 09 20:53:18.615416 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.615390    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-8z2cm"]
Sep 09 20:53:18.615556 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.615519    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.618692 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.618352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-8z2cm"
Sep 09 20:53:18.621713 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.621689    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:53:18.622099 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.621918    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:53:18.622099 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.622042    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:53:18.622099 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.622040    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 20:53:18.622524 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.622504    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 20:53:18.624780 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.624740    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-dvcr7\""
Sep 09 20:53:18.629953 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.629935    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 20:53:18.639570 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.639549    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"]
Sep 09 20:53:18.649256 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.649233    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-8z2cm"]
Sep 09 20:53:18.689737 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.689690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f7307c41-5fb7-4b1c-9ef7-80d5443684ef-klusterlet-config\") pod \"klusterlet-addon-workmgr-664c59fb4c-q7fvq\" (UID: \"f7307c41-5fb7-4b1c-9ef7-80d5443684ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.689737 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.689731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phnz6\" (UniqueName: \"kubernetes.io/projected/c6ebbd15-7d72-4c09-a728-941a45f7af33-kube-api-access-phnz6\") pod \"downloads-85d98c998f-8z2cm\" (UID: \"c6ebbd15-7d72-4c09-a728-941a45f7af33\") " pod="openshift-console/downloads-85d98c998f-8z2cm"
Sep 09 20:53:18.689949 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.689795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7307c41-5fb7-4b1c-9ef7-80d5443684ef-tmp\") pod \"klusterlet-addon-workmgr-664c59fb4c-q7fvq\" (UID: \"f7307c41-5fb7-4b1c-9ef7-80d5443684ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.689949 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.689886    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvpb\" (UniqueName: \"kubernetes.io/projected/f7307c41-5fb7-4b1c-9ef7-80d5443684ef-kube-api-access-xgvpb\") pod \"klusterlet-addon-workmgr-664c59fb4c-q7fvq\" (UID: \"f7307c41-5fb7-4b1c-9ef7-80d5443684ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.791357 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.791318    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f7307c41-5fb7-4b1c-9ef7-80d5443684ef-klusterlet-config\") pod \"klusterlet-addon-workmgr-664c59fb4c-q7fvq\" (UID: \"f7307c41-5fb7-4b1c-9ef7-80d5443684ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.791525 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.791369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnz6\" (UniqueName: \"kubernetes.io/projected/c6ebbd15-7d72-4c09-a728-941a45f7af33-kube-api-access-phnz6\") pod \"downloads-85d98c998f-8z2cm\" (UID: \"c6ebbd15-7d72-4c09-a728-941a45f7af33\") " pod="openshift-console/downloads-85d98c998f-8z2cm"
Sep 09 20:53:18.791525 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.791407    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7307c41-5fb7-4b1c-9ef7-80d5443684ef-tmp\") pod \"klusterlet-addon-workmgr-664c59fb4c-q7fvq\" (UID: \"f7307c41-5fb7-4b1c-9ef7-80d5443684ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.791525 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.791454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvpb\" (UniqueName: \"kubernetes.io/projected/f7307c41-5fb7-4b1c-9ef7-80d5443684ef-kube-api-access-xgvpb\") pod \"klusterlet-addon-workmgr-664c59fb4c-q7fvq\" (UID: \"f7307c41-5fb7-4b1c-9ef7-80d5443684ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.791947 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.791919    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7307c41-5fb7-4b1c-9ef7-80d5443684ef-tmp\") pod \"klusterlet-addon-workmgr-664c59fb4c-q7fvq\" (UID: \"f7307c41-5fb7-4b1c-9ef7-80d5443684ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.794278 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.794258    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/f7307c41-5fb7-4b1c-9ef7-80d5443684ef-klusterlet-config\") pod \"klusterlet-addon-workmgr-664c59fb4c-q7fvq\" (UID: \"f7307c41-5fb7-4b1c-9ef7-80d5443684ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.806597 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.806577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnz6\" (UniqueName: \"kubernetes.io/projected/c6ebbd15-7d72-4c09-a728-941a45f7af33-kube-api-access-phnz6\") pod \"downloads-85d98c998f-8z2cm\" (UID: \"c6ebbd15-7d72-4c09-a728-941a45f7af33\") " pod="openshift-console/downloads-85d98c998f-8z2cm"
Sep 09 20:53:18.814144 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.814115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvpb\" (UniqueName: \"kubernetes.io/projected/f7307c41-5fb7-4b1c-9ef7-80d5443684ef-kube-api-access-xgvpb\") pod \"klusterlet-addon-workmgr-664c59fb4c-q7fvq\" (UID: \"f7307c41-5fb7-4b1c-9ef7-80d5443684ef\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.927811 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.927704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:18.933705 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:18.933673    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-8z2cm"
Sep 09 20:53:19.333234 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.333193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv" event={"ID":"d834aab5-aae4-4735-a1a0-a375d3f37d09","Type":"ContainerStarted","Data":"aa5635f561f495e872980494f281c91f765b18d8a9526abd01be2d69d76744d0"}
Sep 09 20:53:19.358617 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.358569    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-z66wv" podStartSLOduration=33.329442057 podStartE2EDuration="35.358552935s" totalImagesPullingTime="2.029110878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:19.358547669 +0000 UTC m=+88.919574985" watchObservedRunningTime="2026-09-09 20:53:19.358552935 +0000 UTC m=+88.919580251"
Sep 09 20:53:19.392742 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.392715    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"]
Sep 09 20:53:19.398014 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.395899    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-8z2cm"]
Sep 09 20:53:19.725887 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.725853    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"]
Sep 09 20:53:19.731194 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.731174    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"
Sep 09 20:53:19.735887 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.735862    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-55m5d\""
Sep 09 20:53:19.735990 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.735907    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 09 20:53:19.742439 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.742417    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"]
Sep 09 20:53:19.800268 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.800235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/04cd56f3-59a3-4ffa-8318-18a418bbb463-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dlxf9\" (UID: \"04cd56f3-59a3-4ffa-8318-18a418bbb463\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"
Sep 09 20:53:19.901234 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.901197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/04cd56f3-59a3-4ffa-8318-18a418bbb463-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dlxf9\" (UID: \"04cd56f3-59a3-4ffa-8318-18a418bbb463\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"
Sep 09 20:53:19.903976 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:19.903952    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/04cd56f3-59a3-4ffa-8318-18a418bbb463-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-dlxf9\" (UID: \"04cd56f3-59a3-4ffa-8318-18a418bbb463\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"
Sep 09 20:53:20.039563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.039481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"
Sep 09 20:53:20.324375 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.324348    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"]
Sep 09 20:53:20.326730 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:20.326697    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cd56f3_59a3_4ffa_8318_18a418bbb463.slice/crio-ddd3dded82c0f5a8faacf39c4b845e1dd152dfccf4d2c500fe7f605c78c2be09 WatchSource:0}: Error finding container ddd3dded82c0f5a8faacf39c4b845e1dd152dfccf4d2c500fe7f605c78c2be09: Status 404 returned error can't find the container with id ddd3dded82c0f5a8faacf39c4b845e1dd152dfccf4d2c500fe7f605c78c2be09
Sep 09 20:53:20.338143 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.338108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9" event={"ID":"04cd56f3-59a3-4ffa-8318-18a418bbb463","Type":"ContainerStarted","Data":"ddd3dded82c0f5a8faacf39c4b845e1dd152dfccf4d2c500fe7f605c78c2be09"}
Sep 09 20:53:20.340029 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.339533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq" event={"ID":"f7307c41-5fb7-4b1c-9ef7-80d5443684ef","Type":"ContainerStarted","Data":"3381964dea8a08f171bb905fa34937ed413ffbb552a04ea2fdcae945c75528e3"}
Sep 09 20:53:20.341310 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.341260    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-8z2cm" event={"ID":"c6ebbd15-7d72-4c09-a728-941a45f7af33","Type":"ContainerStarted","Data":"e8d5371fff74d1eb0fdd552415a9cbafa01e794d105ecfc3bc8c1332fe7b773e"}
Sep 09 20:53:20.926529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.926498    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-cc4cb7bf5-59kj8"]
Sep 09 20:53:20.933163 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.933138    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:20.935822 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.935398    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:53:20.935822 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.935404    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:53:20.935822 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.935653    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:53:20.936482 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.936446    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:53:20.936659 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.936635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l62dd\""
Sep 09 20:53:20.936796 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.936676    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:53:20.942377 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:20.942356    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc4cb7bf5-59kj8"]
Sep 09 20:53:21.013732 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.013692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-serving-cert\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.013926 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.013832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-service-ca\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.013926 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.013870    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-console-config\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.013926 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.013898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl7c7\" (UniqueName: \"kubernetes.io/projected/03e203a3-7acc-433f-9542-d146deebed37-kube-api-access-vl7c7\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.014112 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.013980    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-oauth-serving-cert\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.014112 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.014040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-oauth-config\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.116542 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.116506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-oauth-config\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.116715 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.116549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-serving-cert\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.116715 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.116594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-service-ca\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.116715 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.116620    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-console-config\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.116715 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.116642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7c7\" (UniqueName: \"kubernetes.io/projected/03e203a3-7acc-433f-9542-d146deebed37-kube-api-access-vl7c7\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.116715 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.116710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-oauth-serving-cert\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.117997 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.117432    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-oauth-serving-cert\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.117997 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.117907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-console-config\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.117997 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.117957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-service-ca\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.120175 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.120147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-serving-cert\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.120676 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.120655    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-oauth-config\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.131485 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.131454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7c7\" (UniqueName: \"kubernetes.io/projected/03e203a3-7acc-433f-9542-d146deebed37-kube-api-access-vl7c7\") pod \"console-cc4cb7bf5-59kj8\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") " pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.246506 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.246428    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:21.521315 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.521236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:53:21.524989 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.524961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1b67f017-f68f-4ced-85c4-01baa0fc1d0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bthvc\" (UID: \"1b67f017-f68f-4ced-85c4-01baa0fc1d0f\") " pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:53:21.760513 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.760477    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5vkbt\""
Sep 09 20:53:21.768672 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:21.768643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bthvc"
Sep 09 20:53:22.081416 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:22.076159    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bthvc"]
Sep 09 20:53:22.084490 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:22.084440    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-cc4cb7bf5-59kj8"]
Sep 09 20:53:23.433863 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:23.433826    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b67f017_f68f_4ced_85c4_01baa0fc1d0f.slice/crio-77a691e7bfe89b09d9decfb179e9b7b540ee929bfba532162cc3e35831b9c7f5 WatchSource:0}: Error finding container 77a691e7bfe89b09d9decfb179e9b7b540ee929bfba532162cc3e35831b9c7f5: Status 404 returned error can't find the container with id 77a691e7bfe89b09d9decfb179e9b7b540ee929bfba532162cc3e35831b9c7f5
Sep 09 20:53:23.434354 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:23.434331    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e203a3_7acc_433f_9542_d146deebed37.slice/crio-3a3c5c4064c5dd9b3b7bf635eef9c49810314d6f3ebdb05a45930783185290c0 WatchSource:0}: Error finding container 3a3c5c4064c5dd9b3b7bf635eef9c49810314d6f3ebdb05a45930783185290c0: Status 404 returned error can't find the container with id 3a3c5c4064c5dd9b3b7bf635eef9c49810314d6f3ebdb05a45930783185290c0
Sep 09 20:53:24.361661 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.361620    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc4cb7bf5-59kj8" event={"ID":"03e203a3-7acc-433f-9542-d146deebed37","Type":"ContainerStarted","Data":"3a3c5c4064c5dd9b3b7bf635eef9c49810314d6f3ebdb05a45930783185290c0"}
Sep 09 20:53:24.363585 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.363514    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bthvc" event={"ID":"1b67f017-f68f-4ced-85c4-01baa0fc1d0f","Type":"ContainerStarted","Data":"d9dbb328a4839ab9fa9c0a663606935e63dfccb35e6bfb9720f92888e210ece3"}
Sep 09 20:53:24.363585 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.363559    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bthvc" event={"ID":"1b67f017-f68f-4ced-85c4-01baa0fc1d0f","Type":"ContainerStarted","Data":"77a691e7bfe89b09d9decfb179e9b7b540ee929bfba532162cc3e35831b9c7f5"}
Sep 09 20:53:24.365324 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.365297    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9" event={"ID":"04cd56f3-59a3-4ffa-8318-18a418bbb463","Type":"ContainerStarted","Data":"4f7b1ffcd38c1a724347f218357613f2d49d0772a01291eec4561a3d2c17c574"}
Sep 09 20:53:24.365711 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.365696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"
Sep 09 20:53:24.368052 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.367996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq" event={"ID":"f7307c41-5fb7-4b1c-9ef7-80d5443684ef","Type":"ContainerStarted","Data":"36aebb9f5a5f5128eeb2cd12d382f2586e1a991386d8d2eba98ab8a9fd2fd25d"}
Sep 09 20:53:24.368729 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.368707    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:24.371138 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.371118    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq"
Sep 09 20:53:24.371999 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.371974    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9"
Sep 09 20:53:24.382451 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.382406    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-dlxf9" podStartSLOduration=1.9316397749999998 podStartE2EDuration="5.382389649s" totalImagesPullingTime="3.450749874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:24.381672385 +0000 UTC m=+93.942699705" watchObservedRunningTime="2026-09-09 20:53:24.382389649 +0000 UTC m=+93.943416964"
Sep 09 20:53:24.401957 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.401892    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-664c59fb4c-q7fvq" podStartSLOduration=2.006735842 podStartE2EDuration="6.401874296s" totalImagesPullingTime="4.395138454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:24.399971052 +0000 UTC m=+93.960998371" watchObservedRunningTime="2026-09-09 20:53:24.401874296 +0000 UTC m=+93.962901615"
Sep 09 20:53:24.767440 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.767410    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rczwq"]
Sep 09 20:53:24.771237 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.771207    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.774252 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.773852    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 09 20:53:24.774252 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.774085    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 09 20:53:24.774955 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.774571    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:53:24.774955 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.774798    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-bf9b4\""
Sep 09 20:53:24.783396 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.782714    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rczwq"]
Sep 09 20:53:24.853078 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.853003    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.853202 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.853078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.853202 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.853136    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.853318 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.853289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwz62\" (UniqueName: \"kubernetes.io/projected/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-kube-api-access-rwz62\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.953877 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.953842    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.954051 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.953887    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwz62\" (UniqueName: \"kubernetes.io/projected/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-kube-api-access-rwz62\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.954051 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.953982    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.954051 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:24.954000    2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 09 20:53:24.954051 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.954033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.954198 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:24.954074    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-prometheus-operator-tls podName:8dc74f72-44b5-49d1-9df3-ed4d978e2eb6 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:25.454053575 +0000 UTC m=+95.015080897 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-prometheus-operator-tls") pod "prometheus-operator-6f4946db56-rczwq" (UID: "8dc74f72-44b5-49d1-9df3-ed4d978e2eb6") : secret "prometheus-operator-tls" not found
Sep 09 20:53:24.954646 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.954622    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.957038 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.957017    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:24.964181 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:24.964162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwz62\" (UniqueName: \"kubernetes.io/projected/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-kube-api-access-rwz62\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:25.375621 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:25.375581    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bthvc" event={"ID":"1b67f017-f68f-4ced-85c4-01baa0fc1d0f","Type":"ContainerStarted","Data":"f4397b07132e42e99dd10ba67a02e36b644e8221db11b85890acbd7493292b91"}
Sep 09 20:53:25.458936 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:25.458903    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:25.461698 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:25.461660    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/8dc74f72-44b5-49d1-9df3-ed4d978e2eb6-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-rczwq\" (UID: \"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:25.707264 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:25.707170    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq"
Sep 09 20:53:27.009887 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:27.009682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-rczwq"]
Sep 09 20:53:27.020090 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:27.019530    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc74f72_44b5_49d1_9df3_ed4d978e2eb6.slice/crio-97da1f95d9038a0ffab256c34b2400406239698db95cc59b92940513b9585838 WatchSource:0}: Error finding container 97da1f95d9038a0ffab256c34b2400406239698db95cc59b92940513b9585838: Status 404 returned error can't find the container with id 97da1f95d9038a0ffab256c34b2400406239698db95cc59b92940513b9585838
Sep 09 20:53:27.384186 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:27.384148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq" event={"ID":"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6","Type":"ContainerStarted","Data":"97da1f95d9038a0ffab256c34b2400406239698db95cc59b92940513b9585838"}
Sep 09 20:53:27.386343 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:27.386309    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc4cb7bf5-59kj8" event={"ID":"03e203a3-7acc-433f-9542-d146deebed37","Type":"ContainerStarted","Data":"674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939"}
Sep 09 20:53:27.408231 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:27.408180    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-cc4cb7bf5-59kj8" podStartSLOduration=4.075709739 podStartE2EDuration="7.408159729s" totalImagesPullingTime="3.33244999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:27.406106317 +0000 UTC m=+96.967133635" watchObservedRunningTime="2026-09-09 20:53:27.408159729 +0000 UTC m=+96.969187046"
Sep 09 20:53:29.202812 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.202742    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:53:29.205871 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.205839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"image-registry-d4ddc64d9-gs6xr\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") " pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:53:29.228876 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.228848    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tn7rg\""
Sep 09 20:53:29.236983 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.236963    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:53:29.265642 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.265612    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xwlqv"
Sep 09 20:53:29.305131 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.305106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:53:29.308364 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.308297    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e815f55d-a4cf-463f-8190-b8fe0f3bd98c-metrics-tls\") pod \"dns-default-fpc6w\" (UID: \"e815f55d-a4cf-463f-8190-b8fe0f3bd98c\") " pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:53:29.406305 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.406265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bthvc" event={"ID":"1b67f017-f68f-4ced-85c4-01baa0fc1d0f","Type":"ContainerStarted","Data":"4f9b8b3b0995e510a55c7e06cc7fcbf560becb513ef0b83b8a040a54a3fea81f"}
Sep 09 20:53:29.408435 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.408410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq" event={"ID":"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6","Type":"ContainerStarted","Data":"2913c122cf075f32d7ccad173f8cf233146ad4c8e966b5aa690dc763c8d96edf"}
Sep 09 20:53:29.408569 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.408460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq" event={"ID":"8dc74f72-44b5-49d1-9df3-ed4d978e2eb6","Type":"ContainerStarted","Data":"0d25b215866ccbb8ad692d5887789ac3275489c45ce929977c33678aed6975ce"}
Sep 09 20:53:29.426609 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.426559    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bthvc" podStartSLOduration=35.9918638 podStartE2EDuration="40.426544642s" totalImagesPullingTime="4.434680842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:52:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:29.424727217 +0000 UTC m=+98.985754533" watchObservedRunningTime="2026-09-09 20:53:29.426544642 +0000 UTC m=+98.987571985"
Sep 09 20:53:29.443999 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.443947    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-rczwq" podStartSLOduration=4.085302389 podStartE2EDuration="5.443927329s" totalImagesPullingTime="1.35862494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:29.442453435 +0000 UTC m=+99.003480751" watchObservedRunningTime="2026-09-09 20:53:29.443927329 +0000 UTC m=+99.004954647"
Sep 09 20:53:29.522814 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.522726    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d4ddc64d9-gs6xr"]
Sep 09 20:53:29.526369 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:29.526341    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb70c45_bc88_4d76_8332_9c53b3a12c72.slice/crio-61bc4ae63188aa2803db41b3542cf10088476db3f02ca45e38d1aca03c63c6e4 WatchSource:0}: Error finding container 61bc4ae63188aa2803db41b3542cf10088476db3f02ca45e38d1aca03c63c6e4: Status 404 returned error can't find the container with id 61bc4ae63188aa2803db41b3542cf10088476db3f02ca45e38d1aca03c63c6e4
Sep 09 20:53:29.567871 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.567848    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ktpgk\""
Sep 09 20:53:29.575682 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.575621    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:53:29.855956 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:29.855928    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fpc6w"]
Sep 09 20:53:29.859187 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:29.859156    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode815f55d_a4cf_463f_8190_b8fe0f3bd98c.slice/crio-29b1d0be41125fc3d779ab050e740e80c757d0780be58c7bc083d34f7d2a2a9d WatchSource:0}: Error finding container 29b1d0be41125fc3d779ab050e740e80c757d0780be58c7bc083d34f7d2a2a9d: Status 404 returned error can't find the container with id 29b1d0be41125fc3d779ab050e740e80c757d0780be58c7bc083d34f7d2a2a9d
Sep 09 20:53:30.413868 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:30.413797    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fpc6w" event={"ID":"e815f55d-a4cf-463f-8190-b8fe0f3bd98c","Type":"ContainerStarted","Data":"29b1d0be41125fc3d779ab050e740e80c757d0780be58c7bc083d34f7d2a2a9d"}
Sep 09 20:53:30.415648 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:30.415610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr" event={"ID":"ecb70c45-bc88-4d76-8332-9c53b3a12c72","Type":"ContainerStarted","Data":"f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96"}
Sep 09 20:53:30.415792 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:30.415661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr" event={"ID":"ecb70c45-bc88-4d76-8332-9c53b3a12c72","Type":"ContainerStarted","Data":"61bc4ae63188aa2803db41b3542cf10088476db3f02ca45e38d1aca03c63c6e4"}
Sep 09 20:53:30.436860 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:30.436797    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr" podStartSLOduration=98.436779551 podStartE2EDuration="1m38.436779551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:30.435309947 +0000 UTC m=+99.996337265" watchObservedRunningTime="2026-09-09 20:53:30.436779551 +0000 UTC m=+99.997806864"
Sep 09 20:53:31.169338 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.169290    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"]
Sep 09 20:53:31.174359 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.174337    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t2xtj"]
Sep 09 20:53:31.174505 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.174472    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.177665 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.177646    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.184688 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.184475    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:53:31.184688 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.184476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 09 20:53:31.185618 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.185600    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:53:31.185618 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.185612    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 09 20:53:31.185817 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.185603    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:53:31.185817 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.185694    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wvj2h\""
Sep 09 20:53:31.185936 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.185823    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zjj4f\""
Sep 09 20:53:31.201586 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.201393    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"]
Sep 09 20:53:31.248113 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.247581    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:31.248113 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.247791    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:31.253174 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.253154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:31.321538 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321506    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-sys\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.321717 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321545    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-wtmp\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.321717 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321568    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.321717 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4985c20-bdee-4ef6-8485-23f67b083942-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.321717 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-textfile\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.321989 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321791    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdjsn\" (UniqueName: \"kubernetes.io/projected/e4985c20-bdee-4ef6-8485-23f67b083942-kube-api-access-cdjsn\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.321989 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321823    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-tls\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.321989 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321890    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.321989 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4985c20-bdee-4ef6-8485-23f67b083942-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.321989 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-metrics-client-ca\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.321989 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.321977    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-root\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.322262 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.322031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4985c20-bdee-4ef6-8485-23f67b083942-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.322262 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.322065    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzg5r\" (UniqueName: \"kubernetes.io/projected/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-kube-api-access-wzg5r\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.419787 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.419657    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:53:31.423615 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.423581    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-sys\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.423939 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.423918    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:53:31.424079 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424004    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-wtmp\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424079 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424197 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424089    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4985c20-bdee-4ef6-8485-23f67b083942-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.424197 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424143    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-textfile\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424304 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-wtmp\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424304 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdjsn\" (UniqueName: \"kubernetes.io/projected/e4985c20-bdee-4ef6-8485-23f67b083942-kube-api-access-cdjsn\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.424304 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424267    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-tls\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424443 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424323    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424443 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4985c20-bdee-4ef6-8485-23f67b083942-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.424443 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424409    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-metrics-client-ca\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424443 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424435    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-sys\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424643 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424460    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-root\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424643 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424497    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4985c20-bdee-4ef6-8485-23f67b083942-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.424643 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424520    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzg5r\" (UniqueName: \"kubernetes.io/projected/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-kube-api-access-wzg5r\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424643 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424627    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-accelerators-collector-config\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424894 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-textfile\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424894 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:31.424823    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:53:31.424894 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.424839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-root\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.424894 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:31.424885    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-tls podName:edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab nodeName:}" failed. No retries permitted until 2026-09-09 20:53:31.924865711 +0000 UTC m=+101.485893009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-tls") pod "node-exporter-t2xtj" (UID: "edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab") : secret "node-exporter-tls" not found
Sep 09 20:53:31.425330 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.425302    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-metrics-client-ca\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.425845 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.425812    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4985c20-bdee-4ef6-8485-23f67b083942-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.428210 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.428187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4985c20-bdee-4ef6-8485-23f67b083942-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.428543 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.428517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.428689 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.428673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4985c20-bdee-4ef6-8485-23f67b083942-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.442404 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.442379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdjsn\" (UniqueName: \"kubernetes.io/projected/e4985c20-bdee-4ef6-8485-23f67b083942-kube-api-access-cdjsn\") pod \"openshift-state-metrics-657fd6c54-pvq5c\" (UID: \"e4985c20-bdee-4ef6-8485-23f67b083942\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.442669 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.442649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzg5r\" (UniqueName: \"kubernetes.io/projected/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-kube-api-access-wzg5r\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.486803 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.486775    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"
Sep 09 20:53:31.928016 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.927981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-tls\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:31.930883 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:31.930858    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab-node-exporter-tls\") pod \"node-exporter-t2xtj\" (UID: \"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab\") " pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:32.093375 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.093345    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t2xtj"
Sep 09 20:53:32.205230 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.204183    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:53:32.225124 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.224333    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.225124 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.224855    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:53:32.227279 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.227257    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:53:32.227488 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.227375    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:53:32.227617 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.227516    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:53:32.227617 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.227598    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:53:32.228006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.227986    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:53:32.228089 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.228063    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:53:32.228210 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.228193    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2nvwd\""
Sep 09 20:53:32.228275 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.228261    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:53:32.228447 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.228427    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:53:32.228953 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.228790    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:53:32.230747 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230612    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.230747 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.230747 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230724    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-web-config\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.230987 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-config-volume\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.230987 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-config-out\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.230987 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230853    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.230987 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.230987 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230898    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.230987 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230943    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.230987 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.230975    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.231263 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.231027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.231263 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.231097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.231263 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.231115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kml2\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-kube-api-access-4kml2\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.331945 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.331905    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.331945 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.331944    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332178 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.331969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332178 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332178 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332039    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kml2\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-kube-api-access-4kml2\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332178 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332178 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332178 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:32.332113    2580 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 09 20:53:32.332178 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-web-config\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332178 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-config-volume\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332547 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:32.332188    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-main-tls podName:32f62432-f9ff-47a8-b878-a808eed7f463 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:32.832172598 +0000 UTC m=+102.393199893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463") : secret "alertmanager-main-tls" not found
Sep 09 20:53:32.332547 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-config-out\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332547 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332235    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332547 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332253    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332547 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332273    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.332547 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.332318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.333073 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:53:32.333052    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-trusted-ca-bundle podName:32f62432-f9ff-47a8-b878-a808eed7f463 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:32.833033373 +0000 UTC m=+102.394060668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463") : configmap references non-existent config key: ca-bundle.crt
Sep 09 20:53:32.334029 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.334004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.335896 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.335573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.335896 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.335621    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.335896 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.335857    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.336067 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.335914    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.336067 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.335954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-config-volume\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.336966 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.336950    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.337222 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.337202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-web-config\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.337804 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.337790    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-config-out\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.341733 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.341715    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kml2\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-kube-api-access-4kml2\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.835713 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.835671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.836196 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.835807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.836657 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.836635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:32.838610 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:32.838582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:33.136777 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:33.136720    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:53:37.206847 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:37.206570    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedaf66b9_6969_4d6c_8fcd_74cc7eecc5ab.slice/crio-53a47f3cd85a54102dde583c8b44480553be19035fb064673aa1ee45d1f8349a WatchSource:0}: Error finding container 53a47f3cd85a54102dde583c8b44480553be19035fb064673aa1ee45d1f8349a: Status 404 returned error can't find the container with id 53a47f3cd85a54102dde583c8b44480553be19035fb064673aa1ee45d1f8349a
Sep 09 20:53:37.447282 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.446352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-8z2cm" event={"ID":"c6ebbd15-7d72-4c09-a728-941a45f7af33","Type":"ContainerStarted","Data":"53512e2f96f3b6450799ae923e717d553a900db64915ce818cbbd776d1d77945"}
Sep 09 20:53:37.447282 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.447172    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-8z2cm"
Sep 09 20:53:37.448833 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.448801    2580 patch_prober.go:28] interesting pod/downloads-85d98c998f-8z2cm container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.15:8080/\": dial tcp 10.132.0.15:8080: connect: connection refused" start-of-body=
Sep 09 20:53:37.448972 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.448858    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-85d98c998f-8z2cm" podUID="c6ebbd15-7d72-4c09-a728-941a45f7af33" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.15:8080/\": dial tcp 10.132.0.15:8080: connect: connection refused"
Sep 09 20:53:37.450559 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.450455    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2xtj" event={"ID":"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab","Type":"ContainerStarted","Data":"53a47f3cd85a54102dde583c8b44480553be19035fb064673aa1ee45d1f8349a"}
Sep 09 20:53:37.471821 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.471698    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-8z2cm" podStartSLOduration=1.569281507 podStartE2EDuration="19.4716783s" totalImagesPullingTime="17.902396793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:37.471131916 +0000 UTC m=+107.032159245" watchObservedRunningTime="2026-09-09 20:53:37.4716783 +0000 UTC m=+107.032705618"
Sep 09 20:53:37.491454 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.491423    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c"]
Sep 09 20:53:37.496427 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:37.496397    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4985c20_bdee_4ef6_8485_23f67b083942.slice/crio-35292801904eb0a3fa820c396090066c4702583b6826ec9c6905fd936b3fe8fd WatchSource:0}: Error finding container 35292801904eb0a3fa820c396090066c4702583b6826ec9c6905fd936b3fe8fd: Status 404 returned error can't find the container with id 35292801904eb0a3fa820c396090066c4702583b6826ec9c6905fd936b3fe8fd
Sep 09 20:53:37.513646 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.513604    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:53:37.520227 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:53:37.520196    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f62432_f9ff_47a8_b878_a808eed7f463.slice/crio-778ac52c17859a7aaf53d0b261a780344ac74289656f897353948db0481950b0 WatchSource:0}: Error finding container 778ac52c17859a7aaf53d0b261a780344ac74289656f897353948db0481950b0: Status 404 returned error can't find the container with id 778ac52c17859a7aaf53d0b261a780344ac74289656f897353948db0481950b0
Sep 09 20:53:37.608232 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.606875    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5dbd4bc7c-c5z79"]
Sep 09 20:53:37.609797 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.609412    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.620550 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.620529    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:53:37.634625 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.634018    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dbd4bc7c-c5z79"]
Sep 09 20:53:37.685565 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.685535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-trusted-ca-bundle\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.685816 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.685794    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-service-ca\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.686236 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.686069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-config\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.686425 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.686406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-serving-cert\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.686576 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.686562    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-oauth-config\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.686724 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.686710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp9tc\" (UniqueName: \"kubernetes.io/projected/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-kube-api-access-mp9tc\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.686904 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.686887    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-oauth-serving-cert\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.789843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.787558    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-oauth-serving-cert\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.789843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.787659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-trusted-ca-bundle\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.789843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.787695    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-service-ca\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.789843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.787738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-config\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.789843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.787824    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-serving-cert\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.789843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.787856    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-oauth-config\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.789843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.787890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp9tc\" (UniqueName: \"kubernetes.io/projected/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-kube-api-access-mp9tc\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.789843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.789051    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-oauth-serving-cert\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.792177 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.790531    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-trusted-ca-bundle\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.792177 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.791202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-service-ca\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.792177 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.791795    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-config\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.799235 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.798837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-serving-cert\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.801313 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.801278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-oauth-config\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.804991 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.804946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp9tc\" (UniqueName: \"kubernetes.io/projected/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-kube-api-access-mp9tc\") pod \"console-5dbd4bc7c-c5z79\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") " pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:37.927540 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:37.927143    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:38.267814 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:38.267610    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5dbd4bc7c-c5z79"]
Sep 09 20:53:38.459529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:38.458915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbd4bc7c-c5z79" event={"ID":"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7","Type":"ContainerStarted","Data":"fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669"}
Sep 09 20:53:38.459529 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:38.458994    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbd4bc7c-c5z79" event={"ID":"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7","Type":"ContainerStarted","Data":"465178cccc45f6f3590e66a9585a398d6f40d472a189675a6583dd3b471ae740"}
Sep 09 20:53:38.463575 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:38.463378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c" event={"ID":"e4985c20-bdee-4ef6-8485-23f67b083942","Type":"ContainerStarted","Data":"f1b07e2ecebb896fe346677086f689449b6c839511ac508faddc9c15739cdf8a"}
Sep 09 20:53:38.463575 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:38.463446    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c" event={"ID":"e4985c20-bdee-4ef6-8485-23f67b083942","Type":"ContainerStarted","Data":"49f7309192a8f859e709c8296ab47fd11735d4bbe4a3a58de9f976b9f3d0cacb"}
Sep 09 20:53:38.463575 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:38.463518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c" event={"ID":"e4985c20-bdee-4ef6-8485-23f67b083942","Type":"ContainerStarted","Data":"35292801904eb0a3fa820c396090066c4702583b6826ec9c6905fd936b3fe8fd"}
Sep 09 20:53:38.467593 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:38.467511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerStarted","Data":"778ac52c17859a7aaf53d0b261a780344ac74289656f897353948db0481950b0"}
Sep 09 20:53:38.471871 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:38.471802    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-8z2cm"
Sep 09 20:53:38.481777 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:38.480023    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5dbd4bc7c-c5z79" podStartSLOduration=1.480007166 podStartE2EDuration="1.480007166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:38.47858879 +0000 UTC m=+108.039616106" watchObservedRunningTime="2026-09-09 20:53:38.480007166 +0000 UTC m=+108.041034484"
Sep 09 20:53:40.499941 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:40.499902    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d4ddc64d9-gs6xr"]
Sep 09 20:53:40.742390 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:40.742356    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dbd4bc7c-c5z79"]
Sep 09 20:53:41.482557 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.482519    2580 generic.go:356] "Generic (PLEG): container finished" podID="32f62432-f9ff-47a8-b878-a808eed7f463" containerID="eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462" exitCode=0
Sep 09 20:53:41.482734 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.482598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerDied","Data":"eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462"}
Sep 09 20:53:41.485177 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.485145    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fpc6w" event={"ID":"e815f55d-a4cf-463f-8190-b8fe0f3bd98c","Type":"ContainerStarted","Data":"5ba515d091a8f23d8a50b04dd70209b970ec799f9a9ac9a16d97d64cf4dc45cb"}
Sep 09 20:53:41.485281 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.485204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fpc6w" event={"ID":"e815f55d-a4cf-463f-8190-b8fe0f3bd98c","Type":"ContainerStarted","Data":"5e7c2deacc1192bfa8dd6921d35e2e0e12a828b5ea55b67bd07b1d200d501db6"}
Sep 09 20:53:41.485539 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.485475    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:53:41.487477 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.487451    2580 generic.go:356] "Generic (PLEG): container finished" podID="edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab" containerID="6b1807ce0a49ba9300cba9520b06ebeaa3fae5303eaa140461f75ce9c8537842" exitCode=0
Sep 09 20:53:41.487575 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.487538    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2xtj" event={"ID":"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab","Type":"ContainerDied","Data":"6b1807ce0a49ba9300cba9520b06ebeaa3fae5303eaa140461f75ce9c8537842"}
Sep 09 20:53:41.492415 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.492377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c" event={"ID":"e4985c20-bdee-4ef6-8485-23f67b083942","Type":"ContainerStarted","Data":"f05b3d33f0fc5811681d1a559392e84bd5ce9414685ee2ade7af1a40e25564c4"}
Sep 09 20:53:41.544080 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.543998    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-pvq5c" podStartSLOduration=7.214902007 podStartE2EDuration="10.543975779s" totalImagesPullingTime="3.329073772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:41.543124358 +0000 UTC m=+111.104151676" watchObservedRunningTime="2026-09-09 20:53:41.543975779 +0000 UTC m=+111.105003099"
Sep 09 20:53:41.565792 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:41.564393    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fpc6w" podStartSLOduration=65.399542739 podStartE2EDuration="1m16.564372917s" totalImagesPullingTime="11.164830178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:41.563278949 +0000 UTC m=+111.124306266" watchObservedRunningTime="2026-09-09 20:53:41.564372917 +0000 UTC m=+111.125400234"
Sep 09 20:53:42.501957 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:42.501915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2xtj" event={"ID":"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab","Type":"ContainerStarted","Data":"9613ade6ecdcfe29c68e71b9b8eec17a1001882874b85cbb0d9633a08b3fadb6"}
Sep 09 20:53:42.502236 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:42.501981    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t2xtj" event={"ID":"edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab","Type":"ContainerStarted","Data":"3fc1f4952df72778ee6d8dcda0886f7dffee8d2967d188ee422f5b68965d2790"}
Sep 09 20:53:42.524687 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:42.524637    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t2xtj" podStartSLOduration=7.624331948 podStartE2EDuration="11.524621855s" totalImagesPullingTime="3.801244673s" totalInitContainerRuntime="99.045234ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:42.5241819 +0000 UTC m=+112.085209218" watchObservedRunningTime="2026-09-09 20:53:42.524621855 +0000 UTC m=+112.085649171"
Sep 09 20:53:44.514258 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:44.514178    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerStarted","Data":"80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee"}
Sep 09 20:53:44.514258 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:44.514243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerStarted","Data":"f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4"}
Sep 09 20:53:44.514802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:44.514284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerStarted","Data":"aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b"}
Sep 09 20:53:44.514802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:44.514326    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerStarted","Data":"04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a"}
Sep 09 20:53:44.514802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:44.514363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerStarted","Data":"17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70"}
Sep 09 20:53:46.526840 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:46.526794    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerStarted","Data":"809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6"}
Sep 09 20:53:46.563226 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:46.563168    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.122301956 podStartE2EDuration="14.563149435s" totalImagesPullingTime="7.404260211s" totalInitContainerRuntime="36.587268ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:46.562968819 +0000 UTC m=+116.123996146" watchObservedRunningTime="2026-09-09 20:53:46.563149435 +0000 UTC m=+116.124176754"
Sep 09 20:53:47.928240 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:47.928205    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:53:50.504919 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:50.504893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:53:51.504011 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:51.503982    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fpc6w"
Sep 09 20:53:51.546649 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:53:51.546616    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc4cb7bf5-59kj8"]
Sep 09 20:54:00.694847 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:00.694803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:54:00.697352 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:00.697329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7b1474e-02e4-4041-a48a-2a3a652388c9-metrics-certs\") pod \"network-metrics-daemon-nzvfm\" (UID: \"b7b1474e-02e4-4041-a48a-2a3a652388c9\") " pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:54:00.752881 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:00.752854    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l2kpj\""
Sep 09 20:54:00.757524 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:00.757507    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-nzvfm"
Sep 09 20:54:01.059279 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:01.059256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-nzvfm"]
Sep 09 20:54:01.061440 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:54:01.061413    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b1474e_02e4_4041_a48a_2a3a652388c9.slice/crio-c6214c05eba80e512658c65644e519d38fd29d984ffdd1f90bb184e04d6c4a79 WatchSource:0}: Error finding container c6214c05eba80e512658c65644e519d38fd29d984ffdd1f90bb184e04d6c4a79: Status 404 returned error can't find the container with id c6214c05eba80e512658c65644e519d38fd29d984ffdd1f90bb184e04d6c4a79
Sep 09 20:54:01.587974 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:01.587938    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nzvfm" event={"ID":"b7b1474e-02e4-4041-a48a-2a3a652388c9","Type":"ContainerStarted","Data":"c6214c05eba80e512658c65644e519d38fd29d984ffdd1f90bb184e04d6c4a79"}
Sep 09 20:54:03.597476 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:03.597387    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nzvfm" event={"ID":"b7b1474e-02e4-4041-a48a-2a3a652388c9","Type":"ContainerStarted","Data":"21bf0e3758fc30097456d81ce711c46f0f9f5f71d94d9b2409d1ff7810f31370"}
Sep 09 20:54:03.597476 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:03.597438    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-nzvfm" event={"ID":"b7b1474e-02e4-4041-a48a-2a3a652388c9","Type":"ContainerStarted","Data":"6fbf8fdaf014fd584037e3f383c3ff34befe6d60f51babbdde39ba35df57794a"}
Sep 09 20:54:03.614817 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:03.614771    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-nzvfm" podStartSLOduration=130.555622182 podStartE2EDuration="2m12.614713055s" totalImagesPullingTime="2.059090873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:03.613320713 +0000 UTC m=+133.174348030" watchObservedRunningTime="2026-09-09 20:54:03.614713055 +0000 UTC m=+133.175740371"
Sep 09 20:54:05.529076 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.529002    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr" podUID="ecb70c45-bc88-4d76-8332-9c53b3a12c72" containerName="registry" containerID="cri-o://f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96" gracePeriod=30
Sep 09 20:54:05.767925 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.767892    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5dbd4bc7c-c5z79" podUID="d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" containerName="console" containerID="cri-o://fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669" gracePeriod=15
Sep 09 20:54:05.772439 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.772419    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:54:05.940190 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.940156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb70c45-bc88-4d76-8332-9c53b3a12c72-ca-trust-extracted\") pod \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") "
Sep 09 20:54:05.940315 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.940206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-installation-pull-secrets\") pod \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") "
Sep 09 20:54:05.940315 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.940261    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-trusted-ca\") pod \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") "
Sep 09 20:54:05.940315 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.940291    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") pod \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") "
Sep 09 20:54:05.940474 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.940335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-image-registry-private-configuration\") pod \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") "
Sep 09 20:54:05.940474 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.940397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6jxc\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-kube-api-access-l6jxc\") pod \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") "
Sep 09 20:54:05.940474 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.940432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-certificates\") pod \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") "
Sep 09 20:54:05.940631 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.940474    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-bound-sa-token\") pod \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\" (UID: \"ecb70c45-bc88-4d76-8332-9c53b3a12c72\") "
Sep 09 20:54:05.941366 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.941191    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ecb70c45-bc88-4d76-8332-9c53b3a12c72" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:05.941743 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.941713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ecb70c45-bc88-4d76-8332-9c53b3a12c72" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:05.948496 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.948441    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ecb70c45-bc88-4d76-8332-9c53b3a12c72" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:05.948634 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.948583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ecb70c45-bc88-4d76-8332-9c53b3a12c72" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:05.950822 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.950689    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ecb70c45-bc88-4d76-8332-9c53b3a12c72" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:05.950822 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.950702    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-kube-api-access-l6jxc" (OuterVolumeSpecName: "kube-api-access-l6jxc") pod "ecb70c45-bc88-4d76-8332-9c53b3a12c72" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72"). InnerVolumeSpecName "kube-api-access-l6jxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:05.950822 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.950729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ecb70c45-bc88-4d76-8332-9c53b3a12c72" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:05.951076 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.951037    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb70c45-bc88-4d76-8332-9c53b3a12c72-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ecb70c45-bc88-4d76-8332-9c53b3a12c72" (UID: "ecb70c45-bc88-4d76-8332-9c53b3a12c72"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:05.997414 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:05.997389    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:54:06.041214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.041137    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6jxc\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-kube-api-access-l6jxc\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.041214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.041162    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-certificates\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.041214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.041172    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-bound-sa-token\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.041214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.041181    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ecb70c45-bc88-4d76-8332-9c53b3a12c72-ca-trust-extracted\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.041214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.041189    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-installation-pull-secrets\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.041214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.041197    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecb70c45-bc88-4d76-8332-9c53b3a12c72-trusted-ca\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.041214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.041205    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ecb70c45-bc88-4d76-8332-9c53b3a12c72-registry-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.041214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.041214    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ecb70c45-bc88-4d76-8332-9c53b3a12c72-image-registry-private-configuration\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.142393 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.142361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-service-ca\") pod \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") "
Sep 09 20:54:06.142552 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.142397    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-serving-cert\") pod \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") "
Sep 09 20:54:06.142552 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.142451    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp9tc\" (UniqueName: \"kubernetes.io/projected/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-kube-api-access-mp9tc\") pod \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") "
Sep 09 20:54:06.142552 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.142471    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-trusted-ca-bundle\") pod \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") "
Sep 09 20:54:06.142680 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.142635    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-config\") pod \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") "
Sep 09 20:54:06.142680 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.142665    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-oauth-serving-cert\") pod \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") "
Sep 09 20:54:06.142807 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.142721    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-oauth-config\") pod \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\" (UID: \"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7\") "
Sep 09 20:54:06.142865 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.142840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-service-ca" (OuterVolumeSpecName: "service-ca") pod "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" (UID: "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:06.143006 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.142976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" (UID: "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:06.143124 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.143035    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-service-ca\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.143124 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.143092    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-config" (OuterVolumeSpecName: "console-config") pod "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" (UID: "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:06.143124 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.143101    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" (UID: "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:06.144988 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.144963    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" (UID: "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:06.144988 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.144976    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" (UID: "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:06.145104 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.145054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-kube-api-access-mp9tc" (OuterVolumeSpecName: "kube-api-access-mp9tc") pod "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" (UID: "d9248fe1-907c-4e66-95ca-7d7c8bee3ef7"). InnerVolumeSpecName "kube-api-access-mp9tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:06.244261 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.244230    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp9tc\" (UniqueName: \"kubernetes.io/projected/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-kube-api-access-mp9tc\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.244261 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.244255    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-trusted-ca-bundle\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.244261 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.244264    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.244481 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.244277    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-oauth-serving-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.244481 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.244285    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-oauth-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.244481 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.244293    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7-console-serving-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:06.610998 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.610964    2580 generic.go:356] "Generic (PLEG): container finished" podID="d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" containerID="fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669" exitCode=0
Sep 09 20:54:06.611414 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.611013    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5dbd4bc7c-c5z79"
Sep 09 20:54:06.611414 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.611052    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbd4bc7c-c5z79" event={"ID":"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7","Type":"ContainerDied","Data":"fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669"}
Sep 09 20:54:06.611414 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.611103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5dbd4bc7c-c5z79" event={"ID":"d9248fe1-907c-4e66-95ca-7d7c8bee3ef7","Type":"ContainerDied","Data":"465178cccc45f6f3590e66a9585a398d6f40d472a189675a6583dd3b471ae740"}
Sep 09 20:54:06.611414 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.611128    2580 scope.go:117] "RemoveContainer" containerID="fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669"
Sep 09 20:54:06.612665 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.612496    2580 generic.go:356] "Generic (PLEG): container finished" podID="ecb70c45-bc88-4d76-8332-9c53b3a12c72" containerID="f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96" exitCode=0
Sep 09 20:54:06.612665 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.612520    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr" event={"ID":"ecb70c45-bc88-4d76-8332-9c53b3a12c72","Type":"ContainerDied","Data":"f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96"}
Sep 09 20:54:06.612665 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.612546    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr"
Sep 09 20:54:06.612665 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.612568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d4ddc64d9-gs6xr" event={"ID":"ecb70c45-bc88-4d76-8332-9c53b3a12c72","Type":"ContainerDied","Data":"61bc4ae63188aa2803db41b3542cf10088476db3f02ca45e38d1aca03c63c6e4"}
Sep 09 20:54:06.620576 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.620557    2580 scope.go:117] "RemoveContainer" containerID="fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669"
Sep 09 20:54:06.620966 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:06.620949    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669\": container with ID starting with fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669 not found: ID does not exist" containerID="fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669"
Sep 09 20:54:06.621011 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.620975    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669"} err="failed to get container status \"fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669\": rpc error: code = NotFound desc = could not find container \"fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669\": container with ID starting with fd0ce5f7387b27a5db1c629e86670664af58b1c77654deb664f3d9ed63a28669 not found: ID does not exist"
Sep 09 20:54:06.621011 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.621003    2580 scope.go:117] "RemoveContainer" containerID="f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96"
Sep 09 20:54:06.628786 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.628752    2580 scope.go:117] "RemoveContainer" containerID="f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96"
Sep 09 20:54:06.629027 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:06.629010    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96\": container with ID starting with f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96 not found: ID does not exist" containerID="f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96"
Sep 09 20:54:06.629074 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.629034    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96"} err="failed to get container status \"f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96\": rpc error: code = NotFound desc = could not find container \"f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96\": container with ID starting with f1e40aa33859d0572a105afb7a4d9578d1b11a56fa6f13f4db7b0b56cc1aeb96 not found: ID does not exist"
Sep 09 20:54:06.634142 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.634121    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5dbd4bc7c-c5z79"]
Sep 09 20:54:06.639701 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.639682    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5dbd4bc7c-c5z79"]
Sep 09 20:54:06.650657 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.650637    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d4ddc64d9-gs6xr"]
Sep 09 20:54:06.654299 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.654280    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d4ddc64d9-gs6xr"]
Sep 09 20:54:06.934849 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.934819    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" path="/var/lib/kubelet/pods/d9248fe1-907c-4e66-95ca-7d7c8bee3ef7/volumes"
Sep 09 20:54:06.935244 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:06.935230    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb70c45-bc88-4d76-8332-9c53b3a12c72" path="/var/lib/kubelet/pods/ecb70c45-bc88-4d76-8332-9c53b3a12c72/volumes"
Sep 09 20:54:16.566776 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:16.566712    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-cc4cb7bf5-59kj8" podUID="03e203a3-7acc-433f-9542-d146deebed37" containerName="console" containerID="cri-o://674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939" gracePeriod=15
Sep 09 20:54:16.857124 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:16.857088    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:54:17.031563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.031537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-oauth-config\") pod \"03e203a3-7acc-433f-9542-d146deebed37\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") "
Sep 09 20:54:17.031747 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.031588    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-oauth-serving-cert\") pod \"03e203a3-7acc-433f-9542-d146deebed37\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") "
Sep 09 20:54:17.031747 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.031612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-service-ca\") pod \"03e203a3-7acc-433f-9542-d146deebed37\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") "
Sep 09 20:54:17.031747 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.031643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-console-config\") pod \"03e203a3-7acc-433f-9542-d146deebed37\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") "
Sep 09 20:54:17.031747 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.031675    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-serving-cert\") pod \"03e203a3-7acc-433f-9542-d146deebed37\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") "
Sep 09 20:54:17.031747 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.031692    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl7c7\" (UniqueName: \"kubernetes.io/projected/03e203a3-7acc-433f-9542-d146deebed37-kube-api-access-vl7c7\") pod \"03e203a3-7acc-433f-9542-d146deebed37\" (UID: \"03e203a3-7acc-433f-9542-d146deebed37\") "
Sep 09 20:54:17.032055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.031973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "03e203a3-7acc-433f-9542-d146deebed37" (UID: "03e203a3-7acc-433f-9542-d146deebed37"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:17.032055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.032031    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-service-ca" (OuterVolumeSpecName: "service-ca") pod "03e203a3-7acc-433f-9542-d146deebed37" (UID: "03e203a3-7acc-433f-9542-d146deebed37"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:17.032055 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.032038    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-console-config" (OuterVolumeSpecName: "console-config") pod "03e203a3-7acc-433f-9542-d146deebed37" (UID: "03e203a3-7acc-433f-9542-d146deebed37"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:17.033958 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.033918    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "03e203a3-7acc-433f-9542-d146deebed37" (UID: "03e203a3-7acc-433f-9542-d146deebed37"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:17.034066 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.033991    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e203a3-7acc-433f-9542-d146deebed37-kube-api-access-vl7c7" (OuterVolumeSpecName: "kube-api-access-vl7c7") pod "03e203a3-7acc-433f-9542-d146deebed37" (UID: "03e203a3-7acc-433f-9542-d146deebed37"). InnerVolumeSpecName "kube-api-access-vl7c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:17.034066 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.033992    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "03e203a3-7acc-433f-9542-d146deebed37" (UID: "03e203a3-7acc-433f-9542-d146deebed37"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:17.132799 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.132747    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-service-ca\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:17.132799 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.132793    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-console-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:17.132982 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.132806    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-serving-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:17.132982 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.132819    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl7c7\" (UniqueName: \"kubernetes.io/projected/03e203a3-7acc-433f-9542-d146deebed37-kube-api-access-vl7c7\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:17.132982 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.132832    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03e203a3-7acc-433f-9542-d146deebed37-console-oauth-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:17.132982 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.132843    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03e203a3-7acc-433f-9542-d146deebed37-oauth-serving-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:17.659775 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.659723    2580 generic.go:356] "Generic (PLEG): container finished" podID="03e203a3-7acc-433f-9542-d146deebed37" containerID="674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939" exitCode=0
Sep 09 20:54:17.660220 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.659790    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-cc4cb7bf5-59kj8"
Sep 09 20:54:17.660220 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.659808    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc4cb7bf5-59kj8" event={"ID":"03e203a3-7acc-433f-9542-d146deebed37","Type":"ContainerDied","Data":"674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939"}
Sep 09 20:54:17.660220 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.659868    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-cc4cb7bf5-59kj8" event={"ID":"03e203a3-7acc-433f-9542-d146deebed37","Type":"ContainerDied","Data":"3a3c5c4064c5dd9b3b7bf635eef9c49810314d6f3ebdb05a45930783185290c0"}
Sep 09 20:54:17.660220 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.659891    2580 scope.go:117] "RemoveContainer" containerID="674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939"
Sep 09 20:54:17.670236 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.670219    2580 scope.go:117] "RemoveContainer" containerID="674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939"
Sep 09 20:54:17.670532 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:17.670515    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939\": container with ID starting with 674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939 not found: ID does not exist" containerID="674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939"
Sep 09 20:54:17.670606 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.670542    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939"} err="failed to get container status \"674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939\": rpc error: code = NotFound desc = could not find container \"674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939\": container with ID starting with 674eaad9cad6723cedb0e6694b0da07ac7d5d5e5551fb618b7af9a1e11458939 not found: ID does not exist"
Sep 09 20:54:17.683727 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.683701    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-cc4cb7bf5-59kj8"]
Sep 09 20:54:17.687574 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:17.687556    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-cc4cb7bf5-59kj8"]
Sep 09 20:54:18.666551 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:18.666470    2580 generic.go:356] "Generic (PLEG): container finished" podID="04dbd6bf-11a8-4786-83d2-6e8df4706cfd" containerID="ccad9ec14ed05ae84dad61121055bbe1e595ee13a7b081af48b28536e578b067" exitCode=0
Sep 09 20:54:18.666551 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:18.666511    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76" event={"ID":"04dbd6bf-11a8-4786-83d2-6e8df4706cfd","Type":"ContainerDied","Data":"ccad9ec14ed05ae84dad61121055bbe1e595ee13a7b081af48b28536e578b067"}
Sep 09 20:54:18.666948 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:18.666839    2580 scope.go:117] "RemoveContainer" containerID="ccad9ec14ed05ae84dad61121055bbe1e595ee13a7b081af48b28536e578b067"
Sep 09 20:54:18.934748 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:18.934661    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03e203a3-7acc-433f-9542-d146deebed37" path="/var/lib/kubelet/pods/03e203a3-7acc-433f-9542-d146deebed37/volumes"
Sep 09 20:54:19.672169 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:19.672135    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-8d7dd4675-mnx76" event={"ID":"04dbd6bf-11a8-4786-83d2-6e8df4706cfd","Type":"ContainerStarted","Data":"ef255be289e577fc52179abc54a035821289fec854740e9c817f9515abbc1213"}
Sep 09 20:54:51.559741 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.559701    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:54:51.560257 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.560142    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="alertmanager" containerID="cri-o://17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70" gracePeriod=120
Sep 09 20:54:51.560257 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.560191    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy-metric" containerID="cri-o://80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee" gracePeriod=120
Sep 09 20:54:51.560257 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.560226    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy-web" containerID="cri-o://aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b" gracePeriod=120
Sep 09 20:54:51.560427 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.560260    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="config-reloader" containerID="cri-o://04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a" gracePeriod=120
Sep 09 20:54:51.560427 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.560279    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy" containerID="cri-o://f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4" gracePeriod=120
Sep 09 20:54:51.560427 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.560260    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="prom-label-proxy" containerID="cri-o://809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6" gracePeriod=120
Sep 09 20:54:51.798464 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798432    2580 generic.go:356] "Generic (PLEG): container finished" podID="32f62432-f9ff-47a8-b878-a808eed7f463" containerID="809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6" exitCode=0
Sep 09 20:54:51.798464 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798459    2580 generic.go:356] "Generic (PLEG): container finished" podID="32f62432-f9ff-47a8-b878-a808eed7f463" containerID="80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee" exitCode=0
Sep 09 20:54:51.798464 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798466    2580 generic.go:356] "Generic (PLEG): container finished" podID="32f62432-f9ff-47a8-b878-a808eed7f463" containerID="f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4" exitCode=0
Sep 09 20:54:51.798464 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798472    2580 generic.go:356] "Generic (PLEG): container finished" podID="32f62432-f9ff-47a8-b878-a808eed7f463" containerID="04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a" exitCode=0
Sep 09 20:54:51.798752 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798478    2580 generic.go:356] "Generic (PLEG): container finished" podID="32f62432-f9ff-47a8-b878-a808eed7f463" containerID="17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70" exitCode=0
Sep 09 20:54:51.798752 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798509    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerDied","Data":"809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6"}
Sep 09 20:54:51.798752 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798578    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerDied","Data":"80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee"}
Sep 09 20:54:51.798752 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798606    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerDied","Data":"f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4"}
Sep 09 20:54:51.798752 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798638    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerDied","Data":"04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a"}
Sep 09 20:54:51.798752 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:51.798677    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerDied","Data":"17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70"}
Sep 09 20:54:52.803327 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.803299    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:52.804946 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.804923    2580 generic.go:356] "Generic (PLEG): container finished" podID="32f62432-f9ff-47a8-b878-a808eed7f463" containerID="aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b" exitCode=0
Sep 09 20:54:52.805056 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.804982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerDied","Data":"aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b"}
Sep 09 20:54:52.805056 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.805035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32f62432-f9ff-47a8-b878-a808eed7f463","Type":"ContainerDied","Data":"778ac52c17859a7aaf53d0b261a780344ac74289656f897353948db0481950b0"}
Sep 09 20:54:52.805126 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.805068    2580 scope.go:117] "RemoveContainer" containerID="809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6"
Sep 09 20:54:52.812268 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.812246    2580 scope.go:117] "RemoveContainer" containerID="80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee"
Sep 09 20:54:52.819038 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.819021    2580 scope.go:117] "RemoveContainer" containerID="f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4"
Sep 09 20:54:52.825642 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.825626    2580 scope.go:117] "RemoveContainer" containerID="aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b"
Sep 09 20:54:52.832209 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.832191    2580 scope.go:117] "RemoveContainer" containerID="04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a"
Sep 09 20:54:52.839468 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.839426    2580 scope.go:117] "RemoveContainer" containerID="17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70"
Sep 09 20:54:52.847507 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.847458    2580 scope.go:117] "RemoveContainer" containerID="eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462"
Sep 09 20:54:52.855644 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.855625    2580 scope.go:117] "RemoveContainer" containerID="809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6"
Sep 09 20:54:52.856011 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:52.855991    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6\": container with ID starting with 809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6 not found: ID does not exist" containerID="809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6"
Sep 09 20:54:52.856110 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.856019    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6"} err="failed to get container status \"809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6\": rpc error: code = NotFound desc = could not find container \"809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6\": container with ID starting with 809ca2b475a29239d5eb18a890ccb31fa819155b40cacf1826f59dbd4d94f4a6 not found: ID does not exist"
Sep 09 20:54:52.856110 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.856037    2580 scope.go:117] "RemoveContainer" containerID="80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee"
Sep 09 20:54:52.856377 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:52.856338    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee\": container with ID starting with 80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee not found: ID does not exist" containerID="80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee"
Sep 09 20:54:52.856434 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.856385    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee"} err="failed to get container status \"80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee\": rpc error: code = NotFound desc = could not find container \"80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee\": container with ID starting with 80dce2e1e22a4584af48e8c7d75c551d4404bd0ecc4c3bb40aab70c2792b69ee not found: ID does not exist"
Sep 09 20:54:52.856434 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.856407    2580 scope.go:117] "RemoveContainer" containerID="f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4"
Sep 09 20:54:52.856673 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:52.856656    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4\": container with ID starting with f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4 not found: ID does not exist" containerID="f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4"
Sep 09 20:54:52.856733 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.856680    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4"} err="failed to get container status \"f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4\": rpc error: code = NotFound desc = could not find container \"f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4\": container with ID starting with f4b41a5a669630609638653830fe0cee63e41d0bdd2f271a6f39793c154914f4 not found: ID does not exist"
Sep 09 20:54:52.856733 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.856695    2580 scope.go:117] "RemoveContainer" containerID="aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b"
Sep 09 20:54:52.860040 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:52.860015    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b\": container with ID starting with aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b not found: ID does not exist" containerID="aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b"
Sep 09 20:54:52.860207 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.860181    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b"} err="failed to get container status \"aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b\": rpc error: code = NotFound desc = could not find container \"aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b\": container with ID starting with aec0d8f3d5685a14b9710e6eb667fe6fb8a88582dc730d9f3957b13cc8475e6b not found: ID does not exist"
Sep 09 20:54:52.860325 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.860306    2580 scope.go:117] "RemoveContainer" containerID="04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a"
Sep 09 20:54:52.860655 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:52.860631    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a\": container with ID starting with 04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a not found: ID does not exist" containerID="04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a"
Sep 09 20:54:52.860731 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.860662    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a"} err="failed to get container status \"04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a\": rpc error: code = NotFound desc = could not find container \"04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a\": container with ID starting with 04fc046c54a204b846de479044628046638fef8418c468feb515d5a7d237ee3a not found: ID does not exist"
Sep 09 20:54:52.860731 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.860684    2580 scope.go:117] "RemoveContainer" containerID="17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70"
Sep 09 20:54:52.861107 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:52.860990    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70\": container with ID starting with 17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70 not found: ID does not exist" containerID="17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70"
Sep 09 20:54:52.861107 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.861020    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70"} err="failed to get container status \"17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70\": rpc error: code = NotFound desc = could not find container \"17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70\": container with ID starting with 17ca902c8b496b96ed2ea0fea3a58c96c836a5d0eb2730637087ea068bcdfa70 not found: ID does not exist"
Sep 09 20:54:52.861107 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.861040    2580 scope.go:117] "RemoveContainer" containerID="eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462"
Sep 09 20:54:52.863227 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:54:52.863184    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462\": container with ID starting with eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462 not found: ID does not exist" containerID="eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462"
Sep 09 20:54:52.863324 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.863230    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462"} err="failed to get container status \"eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462\": rpc error: code = NotFound desc = could not find container \"eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462\": container with ID starting with eafbfb0402c715ff3fadfba6c491ed056c67cfd69c5d1a50b184f65ffa249462 not found: ID does not exist"
Sep 09 20:54:52.905173 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-config-volume\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905173 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-cluster-tls-config\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905173 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-web-config\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905387 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905193    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905387 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905209    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-trusted-ca-bundle\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905387 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905225    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-metric\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905387 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905273    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-main-db\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905387 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905298    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-config-out\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905387 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905338    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kml2\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-kube-api-access-4kml2\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905387 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-metrics-client-ca\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905752 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-web\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905752 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905440    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-tls-assets\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905752 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905471    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-main-tls\") pod \"32f62432-f9ff-47a8-b878-a808eed7f463\" (UID: \"32f62432-f9ff-47a8-b878-a808eed7f463\") "
Sep 09 20:54:52.905929 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.905906    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:52.906719 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.906424    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:52.907263 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.907235    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:52.908264 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.908223    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:52.908685 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.908647    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:52.908986 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.908961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:52.909076 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.908977    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-kube-api-access-4kml2" (OuterVolumeSpecName: "kube-api-access-4kml2") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "kube-api-access-4kml2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:52.909439 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.909408    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-config-volume" (OuterVolumeSpecName: "config-volume") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:52.909549 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.909474    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-config-out" (OuterVolumeSpecName: "config-out") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:54:52.909943 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.909919    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:52.911101 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.911078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:52.912656 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.912630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:52.919877 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:52.919856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-web-config" (OuterVolumeSpecName: "web-config") pod "32f62432-f9ff-47a8-b878-a808eed7f463" (UID: "32f62432-f9ff-47a8-b878-a808eed7f463"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:54:53.006646 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006614    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-main-db\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006646 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006649    2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32f62432-f9ff-47a8-b878-a808eed7f463-config-out\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006663    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kml2\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-kube-api-access-4kml2\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006675    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-metrics-client-ca\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006689    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006702    2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32f62432-f9ff-47a8-b878-a808eed7f463-tls-assets\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006715    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-main-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006727    2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-config-volume\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006741    2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-cluster-tls-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006753    2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-web-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006794    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006810    2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32f62432-f9ff-47a8-b878-a808eed7f463-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.006875 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.006824    2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32f62432-f9ff-47a8-b878-a808eed7f463-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:54:53.809635 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.809608    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:53.836377 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.836348    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:54:53.842649 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.842629    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:54:53.870464 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870436    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:54:53.870715 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870703    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03e203a3-7acc-433f-9542-d146deebed37" containerName="console"
Sep 09 20:54:53.870774 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870718    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="03e203a3-7acc-433f-9542-d146deebed37" containerName="console"
Sep 09 20:54:53.870774 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870727    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy-metric"
Sep 09 20:54:53.870774 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870733    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy-metric"
Sep 09 20:54:53.870774 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870739    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="prom-label-proxy"
Sep 09 20:54:53.870774 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870744    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="prom-label-proxy"
Sep 09 20:54:53.870774 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870752    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="alertmanager"
Sep 09 20:54:53.870774 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870770    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="alertmanager"
Sep 09 20:54:53.870774 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870778    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" containerName="console"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870784    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" containerName="console"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870791    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy-web"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870797    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy-web"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870806    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870811    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870817    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="config-reloader"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870822    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="config-reloader"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870827    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="init-config-reloader"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870832    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="init-config-reloader"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870839    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb70c45-bc88-4d76-8332-9c53b3a12c72" containerName="registry"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb70c45-bc88-4d76-8332-9c53b3a12c72" containerName="registry"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870884    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870890    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy-metric"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870894    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="prom-label-proxy"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870900    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9248fe1-907c-4e66-95ca-7d7c8bee3ef7" containerName="console"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870905    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="03e203a3-7acc-433f-9542-d146deebed37" containerName="console"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870910    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="config-reloader"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870915    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="alertmanager"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870921    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" containerName="kube-rbac-proxy-web"
Sep 09 20:54:53.871003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.870925    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb70c45-bc88-4d76-8332-9c53b3a12c72" containerName="registry"
Sep 09 20:54:53.874696 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.874679    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:53.878856 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.878835    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 09 20:54:53.879464 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.879447    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2nvwd\""
Sep 09 20:54:53.879544 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.879462    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 09 20:54:53.879605 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.879462    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 09 20:54:53.879746 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.879731    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 09 20:54:53.880270 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.880225    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 09 20:54:53.880442 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.880425    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 09 20:54:53.880575 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.880467    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 09 20:54:53.880696 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.880682    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 09 20:54:53.885695 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.885678    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 09 20:54:53.906644 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:53.906585    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:54:54.013629 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013593    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.013843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.013843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.013843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013715    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9bkc\" (UniqueName: \"kubernetes.io/projected/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-kube-api-access-m9bkc\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.013843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.013843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-web-config\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.013843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013810    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.013843 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013837    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.014090 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013857    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.014090 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-config-out\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.014090 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.014090 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.014090 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.013954    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.114482 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.114482 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114484    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9bkc\" (UniqueName: \"kubernetes.io/projected/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-kube-api-access-m9bkc\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.114710 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.114710 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114523    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-web-config\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.114710 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114644    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.114710 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114686    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.114941 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114715    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.114941 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-config-out\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.115049 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114963    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.115049 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.114994    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.115049 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.115023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.115384 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.115061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.115384 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.115110    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.116605 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.116067    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.116605 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.116247    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.116605 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.116553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.118029 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.117969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.118134 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.118081    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.118475 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.118237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.118475 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.118453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-web-config\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.118694 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.118665    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.118795 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.118753    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.118938 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.118912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.119315 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.119293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.120009 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.119994    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-config-out\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.124223 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.124203    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9bkc\" (UniqueName: \"kubernetes.io/projected/c397cbe0-3a68-4254-9e54-e9d0a5c4271f-kube-api-access-m9bkc\") pod \"alertmanager-main-0\" (UID: \"c397cbe0-3a68-4254-9e54-e9d0a5c4271f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.147563 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.147534    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d9c7d68cf-qhjjh"]
Sep 09 20:54:54.150679 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.150660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.155148 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.155131    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 09 20:54:54.155250 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.155131    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 09 20:54:54.156601 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.156584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 09 20:54:54.157031 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.156883    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 09 20:54:54.157207 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.157182    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 09 20:54:54.157630 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.157611    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l62dd\""
Sep 09 20:54:54.161339 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.161319    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 09 20:54:54.177256 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.177238    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d9c7d68cf-qhjjh"]
Sep 09 20:54:54.184177 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.184159    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 09 20:54:54.216145 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.216106    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-console-config\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.216302 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.216243    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-service-ca\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.216302 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.216280    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-oauth-serving-cert\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.216418 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.216315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77w67\" (UniqueName: \"kubernetes.io/projected/79ac51d8-29ca-4a71-a43f-147da2737027-kube-api-access-77w67\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.216418 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.216404    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-oauth-config\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.216498 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.216480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-serving-cert\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.216537 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.216509    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-trusted-ca-bundle\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317029 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-service-ca\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-oauth-serving-cert\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77w67\" (UniqueName: \"kubernetes.io/projected/79ac51d8-29ca-4a71-a43f-147da2737027-kube-api-access-77w67\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317078    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-oauth-config\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317099    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-serving-cert\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317120    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-trusted-ca-bundle\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317214 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317162    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-console-config\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317997 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-service-ca\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317997 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317889    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-console-config\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.317997 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.317944    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-oauth-serving-cert\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.318268 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.318004    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-trusted-ca-bundle\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.319668 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.319639    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-oauth-config\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.319840 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.319822    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-serving-cert\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.327390 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.327369    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77w67\" (UniqueName: \"kubernetes.io/projected/79ac51d8-29ca-4a71-a43f-147da2737027-kube-api-access-77w67\") pod \"console-7d9c7d68cf-qhjjh\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") " pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.460003 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.459916    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:54:54.470562 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.470401    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 09 20:54:54.473444 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:54:54.473418    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc397cbe0_3a68_4254_9e54_e9d0a5c4271f.slice/crio-3e9ad1d1b7a3e46cbe3b5440c273545487c3cde124f198f5fde4c7b30b6294f2 WatchSource:0}: Error finding container 3e9ad1d1b7a3e46cbe3b5440c273545487c3cde124f198f5fde4c7b30b6294f2: Status 404 returned error can't find the container with id 3e9ad1d1b7a3e46cbe3b5440c273545487c3cde124f198f5fde4c7b30b6294f2
Sep 09 20:54:54.738149 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.738078    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d9c7d68cf-qhjjh"]
Sep 09 20:54:54.740908 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:54:54.740880    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ac51d8_29ca_4a71_a43f_147da2737027.slice/crio-be374c08dfa744a38d50d12d58e6ef56c8c1bdf6b0e18015fedca2e689978440 WatchSource:0}: Error finding container be374c08dfa744a38d50d12d58e6ef56c8c1bdf6b0e18015fedca2e689978440: Status 404 returned error can't find the container with id be374c08dfa744a38d50d12d58e6ef56c8c1bdf6b0e18015fedca2e689978440
Sep 09 20:54:54.815203 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.815177    2580 generic.go:356] "Generic (PLEG): container finished" podID="c397cbe0-3a68-4254-9e54-e9d0a5c4271f" containerID="97324a7b2fc9abd30c87e06b490a32de998580ac00c6745d66fba996e51e4ef4" exitCode=0
Sep 09 20:54:54.815578 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.815270    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c397cbe0-3a68-4254-9e54-e9d0a5c4271f","Type":"ContainerDied","Data":"97324a7b2fc9abd30c87e06b490a32de998580ac00c6745d66fba996e51e4ef4"}
Sep 09 20:54:54.815578 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.815358    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c397cbe0-3a68-4254-9e54-e9d0a5c4271f","Type":"ContainerStarted","Data":"3e9ad1d1b7a3e46cbe3b5440c273545487c3cde124f198f5fde4c7b30b6294f2"}
Sep 09 20:54:54.817142 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.817122    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9c7d68cf-qhjjh" event={"ID":"79ac51d8-29ca-4a71-a43f-147da2737027","Type":"ContainerStarted","Data":"75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc"}
Sep 09 20:54:54.817232 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.817163    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9c7d68cf-qhjjh" event={"ID":"79ac51d8-29ca-4a71-a43f-147da2737027","Type":"ContainerStarted","Data":"be374c08dfa744a38d50d12d58e6ef56c8c1bdf6b0e18015fedca2e689978440"}
Sep 09 20:54:54.889186 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.889143    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d9c7d68cf-qhjjh" podStartSLOduration=0.889126276 podStartE2EDuration="889.126276ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:54.887545271 +0000 UTC m=+184.448572587" watchObservedRunningTime="2026-09-09 20:54:54.889126276 +0000 UTC m=+184.450153593"
Sep 09 20:54:54.938234 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:54.938207    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32f62432-f9ff-47a8-b878-a808eed7f463" path="/var/lib/kubelet/pods/32f62432-f9ff-47a8-b878-a808eed7f463/volumes"
Sep 09 20:54:55.825991 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:55.825950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c397cbe0-3a68-4254-9e54-e9d0a5c4271f","Type":"ContainerStarted","Data":"8d8d0f8d05a4f6390f459f64aa9cb983a74a44ba0519e28e0acbed6f9466aee9"}
Sep 09 20:54:55.826384 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:55.826046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c397cbe0-3a68-4254-9e54-e9d0a5c4271f","Type":"ContainerStarted","Data":"776144a135d9be6b6ec941a8a6906ab7ec8517d964e0959b886d626cf7e9b265"}
Sep 09 20:54:55.826384 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:55.826071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c397cbe0-3a68-4254-9e54-e9d0a5c4271f","Type":"ContainerStarted","Data":"bef900a901acb417944512e0c9669c27377aa5c1308b36e4da50a815ec60dfda"}
Sep 09 20:54:55.826384 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:55.826095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c397cbe0-3a68-4254-9e54-e9d0a5c4271f","Type":"ContainerStarted","Data":"ef6e56c760fb02d2c77e42202cef91cab291cd1b700d7f150cf0ac571d59dfb4"}
Sep 09 20:54:55.826384 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:55.826118    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c397cbe0-3a68-4254-9e54-e9d0a5c4271f","Type":"ContainerStarted","Data":"fd1ea9f6424618a8ba2dfaae55dfebf3cb15d7dd4cd5ba33598ee5a20dc5dfe7"}
Sep 09 20:54:55.826384 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:55.826142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c397cbe0-3a68-4254-9e54-e9d0a5c4271f","Type":"ContainerStarted","Data":"2d250300ef79cdcc166167f04d65ea4ee1fff8abb415af31aa8f58dd66cd8763"}
Sep 09 20:54:55.867316 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:54:55.867268    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.851235016 podStartE2EDuration="2.86725155s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.016534ms" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:55.865397585 +0000 UTC m=+185.426424912" watchObservedRunningTime="2026-09-09 20:54:55.86725155 +0000 UTC m=+185.428278866"
Sep 09 20:55:04.460135 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:55:04.460097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:55:04.460584 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:55:04.460166    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:55:04.464887 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:55:04.464865    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:55:04.866740 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:55:04.866714    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:56:21.843234 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:21.843155    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"]
Sep 09 20:56:21.845336 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:21.845320    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:21.847946 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:21.847927    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:56:21.848070 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:21.847932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-4b7ng\""
Sep 09 20:56:21.848070 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:21.847934    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:56:21.858722 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:21.858700    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"]
Sep 09 20:56:21.998937 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:21.998905    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrt9g\" (UniqueName: \"kubernetes.io/projected/c791845c-bd0a-49b5-a601-c822e41c1e04-kube-api-access-lrt9g\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:21.998937 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:21.998946    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:21.999159 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:21.999012    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:22.099943 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:22.099907    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrt9g\" (UniqueName: \"kubernetes.io/projected/c791845c-bd0a-49b5-a601-c822e41c1e04-kube-api-access-lrt9g\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:22.100138 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:22.099975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:22.100138 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:22.100061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:22.100392 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:22.100370    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:22.100494 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:22.100477    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:22.109354 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:22.109329    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrt9g\" (UniqueName: \"kubernetes.io/projected/c791845c-bd0a-49b5-a601-c822e41c1e04-kube-api-access-lrt9g\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:22.154070 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:22.154040    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:22.428544 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:22.428422    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"]
Sep 09 20:56:22.431225 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:56:22.431199    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc791845c_bd0a_49b5_a601_c822e41c1e04.slice/crio-cbd51bc197a3233a629717cf6177bb325fd4c8de0505d82d8344dfd404f47388 WatchSource:0}: Error finding container cbd51bc197a3233a629717cf6177bb325fd4c8de0505d82d8344dfd404f47388: Status 404 returned error can't find the container with id cbd51bc197a3233a629717cf6177bb325fd4c8de0505d82d8344dfd404f47388
Sep 09 20:56:23.161589 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:23.161554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4" event={"ID":"c791845c-bd0a-49b5-a601-c822e41c1e04","Type":"ContainerStarted","Data":"cbd51bc197a3233a629717cf6177bb325fd4c8de0505d82d8344dfd404f47388"}
Sep 09 20:56:29.193633 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:29.193598    2580 generic.go:356] "Generic (PLEG): container finished" podID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerID="a88a98ebb5f277b8172d3d78c45d4197af351bc11230b1d22d09fc00b21b0b83" exitCode=0
Sep 09 20:56:29.194022 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:29.193684    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4" event={"ID":"c791845c-bd0a-49b5-a601-c822e41c1e04","Type":"ContainerDied","Data":"a88a98ebb5f277b8172d3d78c45d4197af351bc11230b1d22d09fc00b21b0b83"}
Sep 09 20:56:32.207943 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:32.207907    2580 generic.go:356] "Generic (PLEG): container finished" podID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerID="97d77e2ebd7964d1a47829be3b56e9382c01cc988f3799f97df57587ed9335d6" exitCode=0
Sep 09 20:56:32.208376 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:32.207990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4" event={"ID":"c791845c-bd0a-49b5-a601-c822e41c1e04","Type":"ContainerDied","Data":"97d77e2ebd7964d1a47829be3b56e9382c01cc988f3799f97df57587ed9335d6"}
Sep 09 20:56:40.243716 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:40.243677    2580 generic.go:356] "Generic (PLEG): container finished" podID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerID="f1aacb874bae442a4ae20d34b4b45c21b57a5b976d8ee54bd0051b8ea062128a" exitCode=0
Sep 09 20:56:40.244130 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:40.243753    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4" event={"ID":"c791845c-bd0a-49b5-a601-c822e41c1e04","Type":"ContainerDied","Data":"f1aacb874bae442a4ae20d34b4b45c21b57a5b976d8ee54bd0051b8ea062128a"}
Sep 09 20:56:41.377611 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.377586    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:41.459856 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.459809    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-util\") pod \"c791845c-bd0a-49b5-a601-c822e41c1e04\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") "
Sep 09 20:56:41.460071 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.459889    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-bundle\") pod \"c791845c-bd0a-49b5-a601-c822e41c1e04\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") "
Sep 09 20:56:41.460071 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.459918    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrt9g\" (UniqueName: \"kubernetes.io/projected/c791845c-bd0a-49b5-a601-c822e41c1e04-kube-api-access-lrt9g\") pod \"c791845c-bd0a-49b5-a601-c822e41c1e04\" (UID: \"c791845c-bd0a-49b5-a601-c822e41c1e04\") "
Sep 09 20:56:41.460519 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.460494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-bundle" (OuterVolumeSpecName: "bundle") pod "c791845c-bd0a-49b5-a601-c822e41c1e04" (UID: "c791845c-bd0a-49b5-a601-c822e41c1e04"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:41.462265 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.462244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c791845c-bd0a-49b5-a601-c822e41c1e04-kube-api-access-lrt9g" (OuterVolumeSpecName: "kube-api-access-lrt9g") pod "c791845c-bd0a-49b5-a601-c822e41c1e04" (UID: "c791845c-bd0a-49b5-a601-c822e41c1e04"). InnerVolumeSpecName "kube-api-access-lrt9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:41.463885 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.463862    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-util" (OuterVolumeSpecName: "util") pod "c791845c-bd0a-49b5-a601-c822e41c1e04" (UID: "c791845c-bd0a-49b5-a601-c822e41c1e04"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:41.560957 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.560868    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-util\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:56:41.560957 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.560903    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c791845c-bd0a-49b5-a601-c822e41c1e04-bundle\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:56:41.560957 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:41.560913    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrt9g\" (UniqueName: \"kubernetes.io/projected/c791845c-bd0a-49b5-a601-c822e41c1e04-kube-api-access-lrt9g\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:56:42.252817 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:42.252786    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4"
Sep 09 20:56:42.252817 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:42.252800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1ab4db4" event={"ID":"c791845c-bd0a-49b5-a601-c822e41c1e04","Type":"ContainerDied","Data":"cbd51bc197a3233a629717cf6177bb325fd4c8de0505d82d8344dfd404f47388"}
Sep 09 20:56:42.253024 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:42.252850    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbd51bc197a3233a629717cf6177bb325fd4c8de0505d82d8344dfd404f47388"
Sep 09 20:56:44.527750 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.527714    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"]
Sep 09 20:56:44.528150 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.528039    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerName="pull"
Sep 09 20:56:44.528150 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.528052    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerName="pull"
Sep 09 20:56:44.528150 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.528062    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerName="util"
Sep 09 20:56:44.528150 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.528068    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerName="util"
Sep 09 20:56:44.528150 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.528081    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerName="extract"
Sep 09 20:56:44.528150 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.528087    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerName="extract"
Sep 09 20:56:44.528368 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.528166    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c791845c-bd0a-49b5-a601-c822e41c1e04" containerName="extract"
Sep 09 20:56:44.559039 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.559012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:56:44.563454 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.563428    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:56:44.563905 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.563877    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:56:44.564053 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.563922    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:56:44.564053 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.563983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-wxbk2\""
Sep 09 20:56:44.564936 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.564909    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"]
Sep 09 20:56:44.584371 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.584346    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b1fcf4da-c795-4a48-b1c6-241312a40b44-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-hkrf8\" (UID: \"b1fcf4da-c795-4a48-b1c6-241312a40b44\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:56:44.584511 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.584414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2q54\" (UniqueName: \"kubernetes.io/projected/b1fcf4da-c795-4a48-b1c6-241312a40b44-kube-api-access-g2q54\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-hkrf8\" (UID: \"b1fcf4da-c795-4a48-b1c6-241312a40b44\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:56:44.685072 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.685032    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b1fcf4da-c795-4a48-b1c6-241312a40b44-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-hkrf8\" (UID: \"b1fcf4da-c795-4a48-b1c6-241312a40b44\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:56:44.685256 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.685114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2q54\" (UniqueName: \"kubernetes.io/projected/b1fcf4da-c795-4a48-b1c6-241312a40b44-kube-api-access-g2q54\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-hkrf8\" (UID: \"b1fcf4da-c795-4a48-b1c6-241312a40b44\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:56:44.687454 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.687423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/b1fcf4da-c795-4a48-b1c6-241312a40b44-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-hkrf8\" (UID: \"b1fcf4da-c795-4a48-b1c6-241312a40b44\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:56:44.697584 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.697561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2q54\" (UniqueName: \"kubernetes.io/projected/b1fcf4da-c795-4a48-b1c6-241312a40b44-kube-api-access-g2q54\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-hkrf8\" (UID: \"b1fcf4da-c795-4a48-b1c6-241312a40b44\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:56:44.870347 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:44.870319    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:56:45.162745 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:45.162653    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"]
Sep 09 20:56:45.168081 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:56:45.168052    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fcf4da_c795_4a48_b1c6_241312a40b44.slice/crio-1d4d4715c2ecd6d23b8c6c0020b013fbee12d77a2a25a2089310304a3be24e4e WatchSource:0}: Error finding container 1d4d4715c2ecd6d23b8c6c0020b013fbee12d77a2a25a2089310304a3be24e4e: Status 404 returned error can't find the container with id 1d4d4715c2ecd6d23b8c6c0020b013fbee12d77a2a25a2089310304a3be24e4e
Sep 09 20:56:45.266117 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:45.266083    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8" event={"ID":"b1fcf4da-c795-4a48-b1c6-241312a40b44","Type":"ContainerStarted","Data":"1d4d4715c2ecd6d23b8c6c0020b013fbee12d77a2a25a2089310304a3be24e4e"}
Sep 09 20:56:48.282128 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:48.282046    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8" event={"ID":"b1fcf4da-c795-4a48-b1c6-241312a40b44","Type":"ContainerStarted","Data":"eb65e730e1ac318143c7096a81d7437c470cdd5ba522bb7b56680664a356e7ad"}
Sep 09 20:56:48.282487 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:48.282189    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:56:49.294129 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.294073    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8" podStartSLOduration=2.527741235 podStartE2EDuration="5.294052584s" totalImagesPullingTime="2.766311349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:48.32634175 +0000 UTC m=+297.887369066" watchObservedRunningTime="2026-09-09 20:56:49.294052584 +0000 UTC m=+298.855079902"
Sep 09 20:56:49.294860 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.294842    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-x4qvf"]
Sep 09 20:56:49.297011 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.296994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.304083 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.304063    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:56:49.304787 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.304743    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zhkn4\""
Sep 09 20:56:49.304787 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.304754    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 09 20:56:49.327958 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.327933    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-certificates\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.328050 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.327960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/00433f90-741b-43e1-a723-2da8ad606be4-cabundle0\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.328095 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.328054    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5r8p\" (UniqueName: \"kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-kube-api-access-x5r8p\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.382646 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.382614    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-x4qvf"]
Sep 09 20:56:49.429187 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.429165    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-certificates\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.429313 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.429192    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/00433f90-741b-43e1-a723-2da8ad606be4-cabundle0\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.429354 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:56:49.429321    2580 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:56:49.429354 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.429334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5r8p\" (UniqueName: \"kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-kube-api-access-x5r8p\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.429421 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:56:49.429339    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:56:49.429421 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:56:49.429403    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-x4qvf: references non-existent secret key: ca.crt
Sep 09 20:56:49.429482 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:56:49.429448    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-certificates podName:00433f90-741b-43e1-a723-2da8ad606be4 nodeName:}" failed. No retries permitted until 2026-09-09 20:56:49.9294327 +0000 UTC m=+299.490460018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-certificates") pod "keda-operator-579c8958f7-x4qvf" (UID: "00433f90-741b-43e1-a723-2da8ad606be4") : references non-existent secret key: ca.crt
Sep 09 20:56:49.429850 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.429834    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/00433f90-741b-43e1-a723-2da8ad606be4-cabundle0\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.452623 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.452594    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5r8p\" (UniqueName: \"kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-kube-api-access-x5r8p\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.933655 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:49.933624    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-certificates\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:49.933852 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:56:49.933755    2580 secret.go:281] references non-existent secret key: ca.crt
Sep 09 20:56:49.933852 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:56:49.933785    2580 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 09 20:56:49.933852 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:56:49.933793    2580 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-x4qvf: references non-existent secret key: ca.crt
Sep 09 20:56:49.933955 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:56:49.933911    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-certificates podName:00433f90-741b-43e1-a723-2da8ad606be4 nodeName:}" failed. No retries permitted until 2026-09-09 20:56:50.933826653 +0000 UTC m=+300.494853952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-certificates") pod "keda-operator-579c8958f7-x4qvf" (UID: "00433f90-741b-43e1-a723-2da8ad606be4") : references non-existent secret key: ca.crt
Sep 09 20:56:50.283625 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.283548    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"]
Sep 09 20:56:50.285851 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.285836    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.292046 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.292024    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:56:50.316184 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.316157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"]
Sep 09 20:56:50.337549 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.337519    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e12159d7-0851-42d4-8e1c-d2146b9bace1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-84h4c\" (UID: \"e12159d7-0851-42d4-8e1c-d2146b9bace1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.337691 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.337577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e12159d7-0851-42d4-8e1c-d2146b9bace1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-84h4c\" (UID: \"e12159d7-0851-42d4-8e1c-d2146b9bace1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.337691 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.337629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7pvn\" (UniqueName: \"kubernetes.io/projected/e12159d7-0851-42d4-8e1c-d2146b9bace1-kube-api-access-p7pvn\") pod \"keda-metrics-apiserver-77857b7b98-84h4c\" (UID: \"e12159d7-0851-42d4-8e1c-d2146b9bace1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.438945 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.438909    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e12159d7-0851-42d4-8e1c-d2146b9bace1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-84h4c\" (UID: \"e12159d7-0851-42d4-8e1c-d2146b9bace1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.439113 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.438969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e12159d7-0851-42d4-8e1c-d2146b9bace1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-84h4c\" (UID: \"e12159d7-0851-42d4-8e1c-d2146b9bace1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.439113 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.439003    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7pvn\" (UniqueName: \"kubernetes.io/projected/e12159d7-0851-42d4-8e1c-d2146b9bace1-kube-api-access-p7pvn\") pod \"keda-metrics-apiserver-77857b7b98-84h4c\" (UID: \"e12159d7-0851-42d4-8e1c-d2146b9bace1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.439336 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.439315    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/e12159d7-0851-42d4-8e1c-d2146b9bace1-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-84h4c\" (UID: \"e12159d7-0851-42d4-8e1c-d2146b9bace1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.441702 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.441684    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/e12159d7-0851-42d4-8e1c-d2146b9bace1-certificates\") pod \"keda-metrics-apiserver-77857b7b98-84h4c\" (UID: \"e12159d7-0851-42d4-8e1c-d2146b9bace1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.451535 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.451504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7pvn\" (UniqueName: \"kubernetes.io/projected/e12159d7-0851-42d4-8e1c-d2146b9bace1-kube-api-access-p7pvn\") pod \"keda-metrics-apiserver-77857b7b98-84h4c\" (UID: \"e12159d7-0851-42d4-8e1c-d2146b9bace1\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.596826 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.596733    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:56:50.836565 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.836533    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 20:56:50.836741 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.836573    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 20:56:50.843713 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.843688    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:56:50.902234 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.902111    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"]
Sep 09 20:56:50.910611 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:56:50.905300    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12159d7_0851_42d4_8e1c_d2146b9bace1.slice/crio-5764ad25d3cc7f8b1b4fec1bc5dce34411aa9273bfd35f6aa979087ed03d2979 WatchSource:0}: Error finding container 5764ad25d3cc7f8b1b4fec1bc5dce34411aa9273bfd35f6aa979087ed03d2979: Status 404 returned error can't find the container with id 5764ad25d3cc7f8b1b4fec1bc5dce34411aa9273bfd35f6aa979087ed03d2979
Sep 09 20:56:50.944034 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.943985    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-certificates\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:50.946713 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:50.946690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/00433f90-741b-43e1-a723-2da8ad606be4-certificates\") pod \"keda-operator-579c8958f7-x4qvf\" (UID: \"00433f90-741b-43e1-a723-2da8ad606be4\") " pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:51.106963 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:51.106925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:56:51.296474 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:51.296430    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c" event={"ID":"e12159d7-0851-42d4-8e1c-d2146b9bace1","Type":"ContainerStarted","Data":"5764ad25d3cc7f8b1b4fec1bc5dce34411aa9273bfd35f6aa979087ed03d2979"}
Sep 09 20:56:51.411267 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:51.411194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-x4qvf"]
Sep 09 20:56:51.420269 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:56:51.420244    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00433f90_741b_43e1_a723_2da8ad606be4.slice/crio-762bbe0a50e80629ecbfaf4e03125e4b2f2ac0e664071b9b5f66f3ad5a08c384 WatchSource:0}: Error finding container 762bbe0a50e80629ecbfaf4e03125e4b2f2ac0e664071b9b5f66f3ad5a08c384: Status 404 returned error can't find the container with id 762bbe0a50e80629ecbfaf4e03125e4b2f2ac0e664071b9b5f66f3ad5a08c384
Sep 09 20:56:52.302776 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:52.302717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-x4qvf" event={"ID":"00433f90-741b-43e1-a723-2da8ad606be4","Type":"ContainerStarted","Data":"762bbe0a50e80629ecbfaf4e03125e4b2f2ac0e664071b9b5f66f3ad5a08c384"}
Sep 09 20:56:54.314849 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:54.314815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c" event={"ID":"e12159d7-0851-42d4-8e1c-d2146b9bace1","Type":"ContainerStarted","Data":"1c683b7dd9001b93fa5f4939c453245833c6cb8cfdb15366f1a1a1ac6e098fe7"}
Sep 09 20:56:54.315296 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:56:54.314916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:57:02.356939 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:57:02.356906    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-x4qvf" event={"ID":"00433f90-741b-43e1-a723-2da8ad606be4","Type":"ContainerStarted","Data":"651844247e9f7fa46cbbd08a923f8ebf64e4c6ac16a99fe18a2cced5b3f36bd8"}
Sep 09 20:57:02.357346 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:57:02.357027    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:57:02.373271 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:57:02.373123    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c" podStartSLOduration=9.711259706 podStartE2EDuration="12.373103883s" totalImagesPullingTime="2.661844177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:56:54.353958836 +0000 UTC m=+303.914986168" watchObservedRunningTime="2026-09-09 20:57:02.373103883 +0000 UTC m=+311.934131202"
Sep 09 20:57:02.373802 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:57:02.373747    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-x4qvf" podStartSLOduration=2.765508948 podStartE2EDuration="13.373733702s" totalImagesPullingTime="10.608224754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:02.372858384 +0000 UTC m=+311.933885703" watchObservedRunningTime="2026-09-09 20:57:02.373733702 +0000 UTC m=+311.934761020"
Sep 09 20:57:05.323961 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:57:05.323929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-84h4c"
Sep 09 20:57:09.289847 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:57:09.289819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-hkrf8"
Sep 09 20:57:23.365088 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:57:23.365054    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-x4qvf"
Sep 09 20:58:16.947257 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:16.947224    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"]
Sep 09 20:58:16.950335 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:16.950317    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:16.960841 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:16.960817    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:58:16.960975 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:16.960871    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:58:16.960975 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:16.960922    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:58:16.961509 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:16.961491    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-wxdr5\""
Sep 09 20:58:16.984160 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:16.984126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"]
Sep 09 20:58:17.022779 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.022734    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-wqfzk"]
Sep 09 20:58:17.025834 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.025814    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:17.029459 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.029437    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-tjdpf\""
Sep 09 20:58:17.029568 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.029506    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 09 20:58:17.063611 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.063580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml7wd\" (UniqueName: \"kubernetes.io/projected/832d2964-8457-4251-9724-69c467c45a4b-kube-api-access-ml7wd\") pod \"llmisvc-controller-manager-645f9c8fd6-wd6sf\" (UID: \"832d2964-8457-4251-9724-69c467c45a4b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:17.063800 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.063638    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/832d2964-8457-4251-9724-69c467c45a4b-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-wd6sf\" (UID: \"832d2964-8457-4251-9724-69c467c45a4b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:17.071997 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.071973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wqfzk"]
Sep 09 20:58:17.164826 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.164797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/efadc29a-a7e9-4adc-81df-2ba658b665c9-data\") pod \"seaweedfs-64568bcd49-wqfzk\" (UID: \"efadc29a-a7e9-4adc-81df-2ba658b665c9\") " pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:17.164826 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.164830    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrfbh\" (UniqueName: \"kubernetes.io/projected/efadc29a-a7e9-4adc-81df-2ba658b665c9-kube-api-access-hrfbh\") pod \"seaweedfs-64568bcd49-wqfzk\" (UID: \"efadc29a-a7e9-4adc-81df-2ba658b665c9\") " pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:17.165044 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.164858    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/832d2964-8457-4251-9724-69c467c45a4b-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-wd6sf\" (UID: \"832d2964-8457-4251-9724-69c467c45a4b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:17.165044 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.164938    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml7wd\" (UniqueName: \"kubernetes.io/projected/832d2964-8457-4251-9724-69c467c45a4b-kube-api-access-ml7wd\") pod \"llmisvc-controller-manager-645f9c8fd6-wd6sf\" (UID: \"832d2964-8457-4251-9724-69c467c45a4b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:17.165044 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:58:17.164958    2580 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 09 20:58:17.165044 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:58:17.165012    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/832d2964-8457-4251-9724-69c467c45a4b-cert podName:832d2964-8457-4251-9724-69c467c45a4b nodeName:}" failed. No retries permitted until 2026-09-09 20:58:17.664995327 +0000 UTC m=+387.226022626 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/832d2964-8457-4251-9724-69c467c45a4b-cert") pod "llmisvc-controller-manager-645f9c8fd6-wd6sf" (UID: "832d2964-8457-4251-9724-69c467c45a4b") : secret "llmisvc-webhook-server-cert" not found
Sep 09 20:58:17.190638 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.190605    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml7wd\" (UniqueName: \"kubernetes.io/projected/832d2964-8457-4251-9724-69c467c45a4b-kube-api-access-ml7wd\") pod \"llmisvc-controller-manager-645f9c8fd6-wd6sf\" (UID: \"832d2964-8457-4251-9724-69c467c45a4b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:17.266269 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.266185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/efadc29a-a7e9-4adc-81df-2ba658b665c9-data\") pod \"seaweedfs-64568bcd49-wqfzk\" (UID: \"efadc29a-a7e9-4adc-81df-2ba658b665c9\") " pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:17.266269 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.266224    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrfbh\" (UniqueName: \"kubernetes.io/projected/efadc29a-a7e9-4adc-81df-2ba658b665c9-kube-api-access-hrfbh\") pod \"seaweedfs-64568bcd49-wqfzk\" (UID: \"efadc29a-a7e9-4adc-81df-2ba658b665c9\") " pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:17.266556 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.266536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/efadc29a-a7e9-4adc-81df-2ba658b665c9-data\") pod \"seaweedfs-64568bcd49-wqfzk\" (UID: \"efadc29a-a7e9-4adc-81df-2ba658b665c9\") " pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:17.276977 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.276946    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrfbh\" (UniqueName: \"kubernetes.io/projected/efadc29a-a7e9-4adc-81df-2ba658b665c9-kube-api-access-hrfbh\") pod \"seaweedfs-64568bcd49-wqfzk\" (UID: \"efadc29a-a7e9-4adc-81df-2ba658b665c9\") " pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:17.334786 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.334741    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:17.614877 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.614857    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-wqfzk"]
Sep 09 20:58:17.617307 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:58:17.617283    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefadc29a_a7e9_4adc_81df_2ba658b665c9.slice/crio-8e89d7087509c371ac242ce024bb283ce7f78b48a1939708322c2ee8530da1b3 WatchSource:0}: Error finding container 8e89d7087509c371ac242ce024bb283ce7f78b48a1939708322c2ee8530da1b3: Status 404 returned error can't find the container with id 8e89d7087509c371ac242ce024bb283ce7f78b48a1939708322c2ee8530da1b3
Sep 09 20:58:17.618575 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.618560    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:58:17.669070 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.669045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/832d2964-8457-4251-9724-69c467c45a4b-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-wd6sf\" (UID: \"832d2964-8457-4251-9724-69c467c45a4b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:17.671438 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.671416    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/832d2964-8457-4251-9724-69c467c45a4b-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-wd6sf\" (UID: \"832d2964-8457-4251-9724-69c467c45a4b\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:17.689973 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.689943    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wqfzk" event={"ID":"efadc29a-a7e9-4adc-81df-2ba658b665c9","Type":"ContainerStarted","Data":"8e89d7087509c371ac242ce024bb283ce7f78b48a1939708322c2ee8530da1b3"}
Sep 09 20:58:17.859942 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:17.859913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:18.143289 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:18.143267    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"]
Sep 09 20:58:18.145658 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:58:18.145631    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod832d2964_8457_4251_9724_69c467c45a4b.slice/crio-4d1c1afb419e73bd776a27c862e8da83f101fae3112763301684016f2cade636 WatchSource:0}: Error finding container 4d1c1afb419e73bd776a27c862e8da83f101fae3112763301684016f2cade636: Status 404 returned error can't find the container with id 4d1c1afb419e73bd776a27c862e8da83f101fae3112763301684016f2cade636
Sep 09 20:58:18.698318 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:18.698278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf" event={"ID":"832d2964-8457-4251-9724-69c467c45a4b","Type":"ContainerStarted","Data":"4d1c1afb419e73bd776a27c862e8da83f101fae3112763301684016f2cade636"}
Sep 09 20:58:21.715906 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:21.715869    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-wqfzk" event={"ID":"efadc29a-a7e9-4adc-81df-2ba658b665c9","Type":"ContainerStarted","Data":"584f8f228867e91fa5b5e85a5742629e00eaea2747c54795e729e47b9e96c01e"}
Sep 09 20:58:21.716380 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:21.715973    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:21.717489 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:21.717460    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf" event={"ID":"832d2964-8457-4251-9724-69c467c45a4b","Type":"ContainerStarted","Data":"276ded3c56867e356a29951cd73212709f5d0e255597fd69e663bd64fbc41740"}
Sep 09 20:58:21.717568 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:21.717521    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:58:21.740506 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:21.740460    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-wqfzk" podStartSLOduration=1.842557821 podStartE2EDuration="5.740446098s" totalImagesPullingTime="3.897888277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:21.738504015 +0000 UTC m=+391.299531332" watchObservedRunningTime="2026-09-09 20:58:21.740446098 +0000 UTC m=+391.301473416"
Sep 09 20:58:21.763616 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:21.763569    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf" podStartSLOduration=2.443443582 podStartE2EDuration="5.76355455s" totalImagesPullingTime="3.320110968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:21.761497521 +0000 UTC m=+391.322524839" watchObservedRunningTime="2026-09-09 20:58:21.76355455 +0000 UTC m=+391.324581867"
Sep 09 20:58:27.725842 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:27.725808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-wqfzk"
Sep 09 20:58:40.571092 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:40.571057    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d9c7d68cf-qhjjh"]
Sep 09 20:58:52.726038 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:58:52.726010    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-wd6sf"
Sep 09 20:59:05.591449 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.591404    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d9c7d68cf-qhjjh" podUID="79ac51d8-29ca-4a71-a43f-147da2737027" containerName="console" containerID="cri-o://75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc" gracePeriod=15
Sep 09 20:59:05.827116 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.827092    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:59:05.918527 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.918494    2580 generic.go:356] "Generic (PLEG): container finished" podID="79ac51d8-29ca-4a71-a43f-147da2737027" containerID="75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc" exitCode=0
Sep 09 20:59:05.918689 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.918537    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d9c7d68cf-qhjjh"
Sep 09 20:59:05.918689 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.918560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9c7d68cf-qhjjh" event={"ID":"79ac51d8-29ca-4a71-a43f-147da2737027","Type":"ContainerDied","Data":"75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc"}
Sep 09 20:59:05.918689 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.918602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d9c7d68cf-qhjjh" event={"ID":"79ac51d8-29ca-4a71-a43f-147da2737027","Type":"ContainerDied","Data":"be374c08dfa744a38d50d12d58e6ef56c8c1bdf6b0e18015fedca2e689978440"}
Sep 09 20:59:05.918689 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.918621    2580 scope.go:117] "RemoveContainer" containerID="75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc"
Sep 09 20:59:05.926771 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.926744    2580 scope.go:117] "RemoveContainer" containerID="75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc"
Sep 09 20:59:05.927066 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:59:05.927048    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc\": container with ID starting with 75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc not found: ID does not exist" containerID="75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc"
Sep 09 20:59:05.927123 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.927076    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc"} err="failed to get container status \"75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc\": rpc error: code = NotFound desc = could not find container \"75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc\": container with ID starting with 75a5a9ed7581197db898f1707e5918281dd9ed4c0b3de406a685b75c3b45abbc not found: ID does not exist"
Sep 09 20:59:05.956957 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.956931    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-serving-cert\") pod \"79ac51d8-29ca-4a71-a43f-147da2737027\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") "
Sep 09 20:59:05.957067 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.956966    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-console-config\") pod \"79ac51d8-29ca-4a71-a43f-147da2737027\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") "
Sep 09 20:59:05.957067 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.956987    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-service-ca\") pod \"79ac51d8-29ca-4a71-a43f-147da2737027\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") "
Sep 09 20:59:05.957156 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.957114    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-trusted-ca-bundle\") pod \"79ac51d8-29ca-4a71-a43f-147da2737027\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") "
Sep 09 20:59:05.957156 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.957143    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-oauth-serving-cert\") pod \"79ac51d8-29ca-4a71-a43f-147da2737027\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") "
Sep 09 20:59:05.957244 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.957161    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77w67\" (UniqueName: \"kubernetes.io/projected/79ac51d8-29ca-4a71-a43f-147da2737027-kube-api-access-77w67\") pod \"79ac51d8-29ca-4a71-a43f-147da2737027\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") "
Sep 09 20:59:05.957244 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.957206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-oauth-config\") pod \"79ac51d8-29ca-4a71-a43f-147da2737027\" (UID: \"79ac51d8-29ca-4a71-a43f-147da2737027\") "
Sep 09 20:59:05.957445 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.957421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-console-config" (OuterVolumeSpecName: "console-config") pod "79ac51d8-29ca-4a71-a43f-147da2737027" (UID: "79ac51d8-29ca-4a71-a43f-147da2737027"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:05.957545 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.957436    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-service-ca" (OuterVolumeSpecName: "service-ca") pod "79ac51d8-29ca-4a71-a43f-147da2737027" (UID: "79ac51d8-29ca-4a71-a43f-147da2737027"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:05.957593 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.957563    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "79ac51d8-29ca-4a71-a43f-147da2737027" (UID: "79ac51d8-29ca-4a71-a43f-147da2737027"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:05.957709 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.957575    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "79ac51d8-29ca-4a71-a43f-147da2737027" (UID: "79ac51d8-29ca-4a71-a43f-147da2737027"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:59:05.964487 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.964317    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "79ac51d8-29ca-4a71-a43f-147da2737027" (UID: "79ac51d8-29ca-4a71-a43f-147da2737027"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:05.964615 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.964510    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "79ac51d8-29ca-4a71-a43f-147da2737027" (UID: "79ac51d8-29ca-4a71-a43f-147da2737027"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:59:05.964615 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:05.964525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ac51d8-29ca-4a71-a43f-147da2737027-kube-api-access-77w67" (OuterVolumeSpecName: "kube-api-access-77w67") pod "79ac51d8-29ca-4a71-a43f-147da2737027" (UID: "79ac51d8-29ca-4a71-a43f-147da2737027"). InnerVolumeSpecName "kube-api-access-77w67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:59:06.058654 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.058618    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-trusted-ca-bundle\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:59:06.058838 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.058668    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-oauth-serving-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:59:06.058838 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.058678    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77w67\" (UniqueName: \"kubernetes.io/projected/79ac51d8-29ca-4a71-a43f-147da2737027-kube-api-access-77w67\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:59:06.058838 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.058687    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-oauth-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:59:06.058838 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.058696    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79ac51d8-29ca-4a71-a43f-147da2737027-console-serving-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:59:06.058838 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.058704    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-console-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:59:06.058838 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.058714    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79ac51d8-29ca-4a71-a43f-147da2737027-service-ca\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 20:59:06.240827 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.240790    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d9c7d68cf-qhjjh"]
Sep 09 20:59:06.242460 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.242439    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d9c7d68cf-qhjjh"]
Sep 09 20:59:06.935160 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:06.935128    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79ac51d8-29ca-4a71-a43f-147da2737027" path="/var/lib/kubelet/pods/79ac51d8-29ca-4a71-a43f-147da2737027/volumes"
Sep 09 20:59:35.105880 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.105848    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-78446"]
Sep 09 20:59:35.106307 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.106168    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79ac51d8-29ca-4a71-a43f-147da2737027" containerName="console"
Sep 09 20:59:35.106307 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.106179    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79ac51d8-29ca-4a71-a43f-147da2737027" containerName="console"
Sep 09 20:59:35.106307 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.106239    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79ac51d8-29ca-4a71-a43f-147da2737027" containerName="console"
Sep 09 20:59:35.109217 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.109197    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:35.111372 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.111354    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 09 20:59:35.112295 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.112271    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-mbv4n\""
Sep 09 20:59:35.113826 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.113808    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-58cfc"]
Sep 09 20:59:35.116995 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.116977    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.119439 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.119417    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 09 20:59:35.119439 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.119430    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 09 20:59:35.119679 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.119666    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-g45tc\""
Sep 09 20:59:35.119739 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.119714    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-78446"]
Sep 09 20:59:35.128977 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.128957    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-58cfc"]
Sep 09 20:59:35.185613 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.185581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2daaf279-d2fe-4f35-bb59-2b38b27bf962-tls-certs\") pod \"model-serving-api-7b954ffc95-78446\" (UID: \"2daaf279-d2fe-4f35-bb59-2b38b27bf962\") " pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:35.185613 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.185613    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb94k\" (UniqueName: \"kubernetes.io/projected/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-kube-api-access-mb94k\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.185847 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.185663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-metrics-certs\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.185847 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.185687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc2w2\" (UniqueName: \"kubernetes.io/projected/2daaf279-d2fe-4f35-bb59-2b38b27bf962-kube-api-access-xc2w2\") pod \"model-serving-api-7b954ffc95-78446\" (UID: \"2daaf279-d2fe-4f35-bb59-2b38b27bf962\") " pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:35.185847 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.185701    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-cert\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.286609 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.286573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2daaf279-d2fe-4f35-bb59-2b38b27bf962-tls-certs\") pod \"model-serving-api-7b954ffc95-78446\" (UID: \"2daaf279-d2fe-4f35-bb59-2b38b27bf962\") " pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:35.286609 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.286610    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb94k\" (UniqueName: \"kubernetes.io/projected/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-kube-api-access-mb94k\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.286873 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:59:35.286719    2580 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 09 20:59:35.286873 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.286741    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-metrics-certs\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.286873 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:59:35.286824    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2daaf279-d2fe-4f35-bb59-2b38b27bf962-tls-certs podName:2daaf279-d2fe-4f35-bb59-2b38b27bf962 nodeName:}" failed. No retries permitted until 2026-09-09 20:59:35.786802274 +0000 UTC m=+465.347829569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2daaf279-d2fe-4f35-bb59-2b38b27bf962-tls-certs") pod "model-serving-api-7b954ffc95-78446" (UID: "2daaf279-d2fe-4f35-bb59-2b38b27bf962") : secret "model-serving-api-tls" not found
Sep 09 20:59:35.286873 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.286848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc2w2\" (UniqueName: \"kubernetes.io/projected/2daaf279-d2fe-4f35-bb59-2b38b27bf962-kube-api-access-xc2w2\") pod \"model-serving-api-7b954ffc95-78446\" (UID: \"2daaf279-d2fe-4f35-bb59-2b38b27bf962\") " pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:35.286873 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.286869    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-cert\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.287079 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:59:35.286883    2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 09 20:59:35.287079 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:59:35.286944    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-metrics-certs podName:7e4947fe-03a9-4b6c-889e-3bc9c3d3a401 nodeName:}" failed. No retries permitted until 2026-09-09 20:59:35.786925125 +0000 UTC m=+465.347952443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-metrics-certs") pod "odh-model-controller-7b966d7984-58cfc" (UID: "7e4947fe-03a9-4b6c-889e-3bc9c3d3a401") : secret "odh-model-controller-metrics-tls" not found
Sep 09 20:59:35.287079 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:59:35.287008    2580 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 09 20:59:35.287079 ip-10-0-130-40 kubenswrapper[2580]: E0909 20:59:35.287052    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-cert podName:7e4947fe-03a9-4b6c-889e-3bc9c3d3a401 nodeName:}" failed. No retries permitted until 2026-09-09 20:59:35.787043425 +0000 UTC m=+465.348070753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-cert") pod "odh-model-controller-7b966d7984-58cfc" (UID: "7e4947fe-03a9-4b6c-889e-3bc9c3d3a401") : secret "odh-model-controller-webhook-cert" not found
Sep 09 20:59:35.295893 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.295870    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb94k\" (UniqueName: \"kubernetes.io/projected/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-kube-api-access-mb94k\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.296049 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.296032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc2w2\" (UniqueName: \"kubernetes.io/projected/2daaf279-d2fe-4f35-bb59-2b38b27bf962-kube-api-access-xc2w2\") pod \"model-serving-api-7b954ffc95-78446\" (UID: \"2daaf279-d2fe-4f35-bb59-2b38b27bf962\") " pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:35.792773 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.792733    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-metrics-certs\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.792975 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.792803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-cert\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.792975 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.792874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2daaf279-d2fe-4f35-bb59-2b38b27bf962-tls-certs\") pod \"model-serving-api-7b954ffc95-78446\" (UID: \"2daaf279-d2fe-4f35-bb59-2b38b27bf962\") " pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:35.795280 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.795249    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2daaf279-d2fe-4f35-bb59-2b38b27bf962-tls-certs\") pod \"model-serving-api-7b954ffc95-78446\" (UID: \"2daaf279-d2fe-4f35-bb59-2b38b27bf962\") " pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:35.795414 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.795311    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-metrics-certs\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:35.795414 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:35.795355    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7e4947fe-03a9-4b6c-889e-3bc9c3d3a401-cert\") pod \"odh-model-controller-7b966d7984-58cfc\" (UID: \"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401\") " pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:36.019853 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:36.019820    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:36.030901 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:36.030876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:36.302346 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:36.302320    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-78446"]
Sep 09 20:59:36.305217 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:59:36.305188    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2daaf279_d2fe_4f35_bb59_2b38b27bf962.slice/crio-82db4a98b51c7299d15f878b0830200269a89a1d0628755f5a810c3da13fb26b WatchSource:0}: Error finding container 82db4a98b51c7299d15f878b0830200269a89a1d0628755f5a810c3da13fb26b: Status 404 returned error can't find the container with id 82db4a98b51c7299d15f878b0830200269a89a1d0628755f5a810c3da13fb26b
Sep 09 20:59:36.318569 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:36.318545    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-58cfc"]
Sep 09 20:59:36.320897 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:59:36.320874    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4947fe_03a9_4b6c_889e_3bc9c3d3a401.slice/crio-729f18f55623f9f990ff9302b4e668b1c192c4f9630423dd7e889faac1d15e84 WatchSource:0}: Error finding container 729f18f55623f9f990ff9302b4e668b1c192c4f9630423dd7e889faac1d15e84: Status 404 returned error can't find the container with id 729f18f55623f9f990ff9302b4e668b1c192c4f9630423dd7e889faac1d15e84
Sep 09 20:59:37.060053 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:37.060008    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-58cfc" event={"ID":"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401","Type":"ContainerStarted","Data":"729f18f55623f9f990ff9302b4e668b1c192c4f9630423dd7e889faac1d15e84"}
Sep 09 20:59:37.062363 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:37.062299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-78446" event={"ID":"2daaf279-d2fe-4f35-bb59-2b38b27bf962","Type":"ContainerStarted","Data":"82db4a98b51c7299d15f878b0830200269a89a1d0628755f5a810c3da13fb26b"}
Sep 09 20:59:40.084614 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:40.084576    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-58cfc" event={"ID":"7e4947fe-03a9-4b6c-889e-3bc9c3d3a401","Type":"ContainerStarted","Data":"35bef77e271ee53d2fa9d21dec07e508c175dace1e982970a1df54f32b386060"}
Sep 09 20:59:40.085104 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:40.084640    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:40.086202 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:40.086182    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-78446" event={"ID":"2daaf279-d2fe-4f35-bb59-2b38b27bf962","Type":"ContainerStarted","Data":"a95dd4df64e3a183cc115c3d33910f797c2a425e1fa040903ad5fa7a86057603"}
Sep 09 20:59:40.086289 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:40.086240    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:40.104989 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:40.104938    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-58cfc" podStartSLOduration=2.268578592 podStartE2EDuration="5.104919195s" totalImagesPullingTime="2.836340603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:40.104542687 +0000 UTC m=+469.665570003" watchObservedRunningTime="2026-09-09 20:59:40.104919195 +0000 UTC m=+469.665946512"
Sep 09 20:59:40.130512 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:40.130463    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-78446" podStartSLOduration=2.327947063 podStartE2EDuration="5.13044913s" totalImagesPullingTime="2.802502067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:59:40.127478798 +0000 UTC m=+469.688506126" watchObservedRunningTime="2026-09-09 20:59:40.13044913 +0000 UTC m=+469.691476448"
Sep 09 20:59:51.093149 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:51.093123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-58cfc"
Sep 09 20:59:51.095150 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:51.095129    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-78446"
Sep 09 20:59:52.098978 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.098943    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-pgtx9"]
Sep 09 20:59:52.103377 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.103359    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pgtx9"
Sep 09 20:59:52.106096 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.106081    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 20:59:52.115702 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.115682    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pgtx9"]
Sep 09 20:59:52.240988 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.240957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-model-cache\") pod \"s3-init-pgtx9\" (UID: \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\") " pod="kserve/s3-init-pgtx9"
Sep 09 20:59:52.241191 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.241026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cltdv\" (UniqueName: \"kubernetes.io/projected/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-kube-api-access-cltdv\") pod \"s3-init-pgtx9\" (UID: \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\") " pod="kserve/s3-init-pgtx9"
Sep 09 20:59:52.342074 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.342036    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-model-cache\") pod \"s3-init-pgtx9\" (UID: \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\") " pod="kserve/s3-init-pgtx9"
Sep 09 20:59:52.342245 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.342095    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cltdv\" (UniqueName: \"kubernetes.io/projected/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-kube-api-access-cltdv\") pod \"s3-init-pgtx9\" (UID: \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\") " pod="kserve/s3-init-pgtx9"
Sep 09 20:59:52.342421 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.342403    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-model-cache\") pod \"s3-init-pgtx9\" (UID: \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\") " pod="kserve/s3-init-pgtx9"
Sep 09 20:59:52.351159 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.351102    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cltdv\" (UniqueName: \"kubernetes.io/projected/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-kube-api-access-cltdv\") pod \"s3-init-pgtx9\" (UID: \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\") " pod="kserve/s3-init-pgtx9"
Sep 09 20:59:52.426723 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.426690    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pgtx9"
Sep 09 20:59:52.710887 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:52.710863    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-pgtx9"]
Sep 09 20:59:52.712949 ip-10-0-130-40 kubenswrapper[2580]: W0909 20:59:52.712918    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode99f790f_82a3_4ef5_9231_31ea5a96f1b0.slice/crio-3d92f5cfa84c83db9e081247dbfac26f93fa5f9fabd44e357e0fefb20ffd719c WatchSource:0}: Error finding container 3d92f5cfa84c83db9e081247dbfac26f93fa5f9fabd44e357e0fefb20ffd719c: Status 404 returned error can't find the container with id 3d92f5cfa84c83db9e081247dbfac26f93fa5f9fabd44e357e0fefb20ffd719c
Sep 09 20:59:53.153057 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:53.153024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pgtx9" event={"ID":"e99f790f-82a3-4ef5-9231-31ea5a96f1b0","Type":"ContainerStarted","Data":"3d92f5cfa84c83db9e081247dbfac26f93fa5f9fabd44e357e0fefb20ffd719c"}
Sep 09 20:59:57.181473 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:57.181392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pgtx9" event={"ID":"e99f790f-82a3-4ef5-9231-31ea5a96f1b0","Type":"ContainerStarted","Data":"975cb1ce335c5b12c9c94148d7377df9548f7e77a4194c6391db2608943bec46"}
Sep 09 20:59:59.192694 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:59.192662    2580 generic.go:356] "Generic (PLEG): container finished" podID="e99f790f-82a3-4ef5-9231-31ea5a96f1b0" containerID="975cb1ce335c5b12c9c94148d7377df9548f7e77a4194c6391db2608943bec46" exitCode=0
Sep 09 20:59:59.193091 ip-10-0-130-40 kubenswrapper[2580]: I0909 20:59:59.192735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pgtx9" event={"ID":"e99f790f-82a3-4ef5-9231-31ea5a96f1b0","Type":"ContainerDied","Data":"975cb1ce335c5b12c9c94148d7377df9548f7e77a4194c6391db2608943bec46"}
Sep 09 21:00:04.221475 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:00:04.221437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pgtx9" event={"ID":"e99f790f-82a3-4ef5-9231-31ea5a96f1b0","Type":"ContainerStarted","Data":"87db541da3bc89a51f65b5f59926e1080858e0d480490621dc6be82b7ab8114e"}
Sep 09 21:00:04.237575 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:00:04.237518    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-pgtx9" podStartSLOduration=1.782754491 podStartE2EDuration="12.237495487s" totalImagesPullingTime="8.645151489s" totalInitContainerRuntime="1.809589507s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:59:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:04.23727175 +0000 UTC m=+493.798299067" watchObservedRunningTime="2026-09-09 21:00:04.237495487 +0000 UTC m=+493.798522807"
Sep 09 21:01:09.517651 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:09.517616    2580 generic.go:356] "Generic (PLEG): container finished" podID="e99f790f-82a3-4ef5-9231-31ea5a96f1b0" containerID="87db541da3bc89a51f65b5f59926e1080858e0d480490621dc6be82b7ab8114e" exitCode=0
Sep 09 21:01:09.518080 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:09.517685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pgtx9" event={"ID":"e99f790f-82a3-4ef5-9231-31ea5a96f1b0","Type":"ContainerDied","Data":"87db541da3bc89a51f65b5f59926e1080858e0d480490621dc6be82b7ab8114e"}
Sep 09 21:01:10.652200 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:10.652179    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pgtx9"
Sep 09 21:01:10.801692 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:10.801604    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cltdv\" (UniqueName: \"kubernetes.io/projected/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-kube-api-access-cltdv\") pod \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\" (UID: \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\") "
Sep 09 21:01:10.801896 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:10.801718    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-model-cache\") pod \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\" (UID: \"e99f790f-82a3-4ef5-9231-31ea5a96f1b0\") "
Sep 09 21:01:10.803911 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:10.803871    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-kube-api-access-cltdv" (OuterVolumeSpecName: "kube-api-access-cltdv") pod "e99f790f-82a3-4ef5-9231-31ea5a96f1b0" (UID: "e99f790f-82a3-4ef5-9231-31ea5a96f1b0"). InnerVolumeSpecName "kube-api-access-cltdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:10.820678 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:10.820630    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e99f790f-82a3-4ef5-9231-31ea5a96f1b0" (UID: "e99f790f-82a3-4ef5-9231-31ea5a96f1b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:10.902461 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:10.902416    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-model-cache\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:01:10.902461 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:10.902459    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cltdv\" (UniqueName: \"kubernetes.io/projected/e99f790f-82a3-4ef5-9231-31ea5a96f1b0-kube-api-access-cltdv\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:01:11.527490 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:11.527459    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-pgtx9"
Sep 09 21:01:11.527638 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:11.527457    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-pgtx9" event={"ID":"e99f790f-82a3-4ef5-9231-31ea5a96f1b0","Type":"ContainerDied","Data":"3d92f5cfa84c83db9e081247dbfac26f93fa5f9fabd44e357e0fefb20ffd719c"}
Sep 09 21:01:11.527638 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:11.527570    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d92f5cfa84c83db9e081247dbfac26f93fa5f9fabd44e357e0fefb20ffd719c"
Sep 09 21:01:12.236131 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.236103    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"]
Sep 09 21:01:12.236487 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.236419    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99f790f-82a3-4ef5-9231-31ea5a96f1b0" containerName="download-hf-model"
Sep 09 21:01:12.236487 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.236431    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99f790f-82a3-4ef5-9231-31ea5a96f1b0" containerName="download-hf-model"
Sep 09 21:01:12.236487 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.236450    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e99f790f-82a3-4ef5-9231-31ea5a96f1b0" containerName="s3-init"
Sep 09 21:01:12.236487 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.236455    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e99f790f-82a3-4ef5-9231-31ea5a96f1b0" containerName="s3-init"
Sep 09 21:01:12.236614 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.236511    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e99f790f-82a3-4ef5-9231-31ea5a96f1b0" containerName="s3-init"
Sep 09 21:01:12.238675 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.238661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:12.240787 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.240751    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 09 21:01:12.248175 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.248155    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"]
Sep 09 21:01:12.313318 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.313277    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpkd6\" (UniqueName: \"kubernetes.io/projected/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-kube-api-access-vpkd6\") pod \"seaweedfs-tls-custom-6644bf5d6c-zxbdf\" (UID: \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:12.313482 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.313367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-zxbdf\" (UID: \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:12.414552 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.414521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-zxbdf\" (UID: \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:12.414708 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.414589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpkd6\" (UniqueName: \"kubernetes.io/projected/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-kube-api-access-vpkd6\") pod \"seaweedfs-tls-custom-6644bf5d6c-zxbdf\" (UID: \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:12.414881 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.414861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-zxbdf\" (UID: \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:12.429345 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.429317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpkd6\" (UniqueName: \"kubernetes.io/projected/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-kube-api-access-vpkd6\") pod \"seaweedfs-tls-custom-6644bf5d6c-zxbdf\" (UID: \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:12.548590 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.548513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:12.818324 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:12.818294    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"]
Sep 09 21:01:12.821028 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:01:12.821004    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7eb553b_3a59_4616_b8b5_6ff7950c95b1.slice/crio-99dc32c7665012cfbc1883d8bf3872d242533313708121134d9af8d613991663 WatchSource:0}: Error finding container 99dc32c7665012cfbc1883d8bf3872d242533313708121134d9af8d613991663: Status 404 returned error can't find the container with id 99dc32c7665012cfbc1883d8bf3872d242533313708121134d9af8d613991663
Sep 09 21:01:13.539116 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:13.539080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf" event={"ID":"e7eb553b-3a59-4616-b8b5-6ff7950c95b1","Type":"ContainerStarted","Data":"90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691"}
Sep 09 21:01:13.539579 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:13.539140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf" event={"ID":"e7eb553b-3a59-4616-b8b5-6ff7950c95b1","Type":"ContainerStarted","Data":"99dc32c7665012cfbc1883d8bf3872d242533313708121134d9af8d613991663"}
Sep 09 21:01:13.557092 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:13.557039    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf" podStartSLOduration=1.28191872 podStartE2EDuration="1.557023319s" totalImagesPullingTime="275.104599ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:13.554743182 +0000 UTC m=+563.115770499" watchObservedRunningTime="2026-09-09 21:01:13.557023319 +0000 UTC m=+563.118050637"
Sep 09 21:01:14.814403 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:14.814367    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"]
Sep 09 21:01:15.548326 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:15.548289    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf" podUID="e7eb553b-3a59-4616-b8b5-6ff7950c95b1" containerName="seaweedfs-tls-custom" containerID="cri-o://90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691" gracePeriod=30
Sep 09 21:01:16.792221 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:16.792198    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:16.845675 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:16.845591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpkd6\" (UniqueName: \"kubernetes.io/projected/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-kube-api-access-vpkd6\") pod \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\" (UID: \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\") "
Sep 09 21:01:16.845675 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:16.845631    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-data\") pod \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\" (UID: \"e7eb553b-3a59-4616-b8b5-6ff7950c95b1\") "
Sep 09 21:01:16.846578 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:16.846551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-data" (OuterVolumeSpecName: "data") pod "e7eb553b-3a59-4616-b8b5-6ff7950c95b1" (UID: "e7eb553b-3a59-4616-b8b5-6ff7950c95b1"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:16.847859 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:16.847836    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-kube-api-access-vpkd6" (OuterVolumeSpecName: "kube-api-access-vpkd6") pod "e7eb553b-3a59-4616-b8b5-6ff7950c95b1" (UID: "e7eb553b-3a59-4616-b8b5-6ff7950c95b1"). InnerVolumeSpecName "kube-api-access-vpkd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:16.946107 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:16.946082    2580 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-data\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:01:16.946107 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:16.946108    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpkd6\" (UniqueName: \"kubernetes.io/projected/e7eb553b-3a59-4616-b8b5-6ff7950c95b1-kube-api-access-vpkd6\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:01:17.560539 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.560503    2580 generic.go:356] "Generic (PLEG): container finished" podID="e7eb553b-3a59-4616-b8b5-6ff7950c95b1" containerID="90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691" exitCode=0
Sep 09 21:01:17.560705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.560553    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"
Sep 09 21:01:17.560705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.560592    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf" event={"ID":"e7eb553b-3a59-4616-b8b5-6ff7950c95b1","Type":"ContainerDied","Data":"90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691"}
Sep 09 21:01:17.560705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.560643    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf" event={"ID":"e7eb553b-3a59-4616-b8b5-6ff7950c95b1","Type":"ContainerDied","Data":"99dc32c7665012cfbc1883d8bf3872d242533313708121134d9af8d613991663"}
Sep 09 21:01:17.560705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.560664    2580 scope.go:117] "RemoveContainer" containerID="90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691"
Sep 09 21:01:17.569918 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.569897    2580 scope.go:117] "RemoveContainer" containerID="90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691"
Sep 09 21:01:17.570157 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:01:17.570138    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691\": container with ID starting with 90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691 not found: ID does not exist" containerID="90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691"
Sep 09 21:01:17.570213 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.570171    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691"} err="failed to get container status \"90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691\": rpc error: code = NotFound desc = could not find container \"90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691\": container with ID starting with 90b08707172d6dea1abcbb180e236f3752ae50951a1b5dbe9192ed2146416691 not found: ID does not exist"
Sep 09 21:01:17.577898 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.577867    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"]
Sep 09 21:01:17.579608 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.579583    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-zxbdf"]
Sep 09 21:01:17.613819 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.613789    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-sb57f"]
Sep 09 21:01:17.614149 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.614137    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7eb553b-3a59-4616-b8b5-6ff7950c95b1" containerName="seaweedfs-tls-custom"
Sep 09 21:01:17.614197 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.614152    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7eb553b-3a59-4616-b8b5-6ff7950c95b1" containerName="seaweedfs-tls-custom"
Sep 09 21:01:17.614231 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.614218    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7eb553b-3a59-4616-b8b5-6ff7950c95b1" containerName="seaweedfs-tls-custom"
Sep 09 21:01:17.617355 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.617341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.620332 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.620304    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 09 21:01:17.620542 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.620526    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 09 21:01:17.627378 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.627358    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-sb57f"]
Sep 09 21:01:17.651386 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.651353    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znwqw\" (UniqueName: \"kubernetes.io/projected/c2f94f01-0623-4a8b-8fb0-309ca75d6725-kube-api-access-znwqw\") pod \"seaweedfs-tls-custom-8b44d767d-sb57f\" (UID: \"c2f94f01-0623-4a8b-8fb0-309ca75d6725\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.651535 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.651409    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c2f94f01-0623-4a8b-8fb0-309ca75d6725-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-sb57f\" (UID: \"c2f94f01-0623-4a8b-8fb0-309ca75d6725\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.651535 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.651461    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c2f94f01-0623-4a8b-8fb0-309ca75d6725-data\") pod \"seaweedfs-tls-custom-8b44d767d-sb57f\" (UID: \"c2f94f01-0623-4a8b-8fb0-309ca75d6725\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.751964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.751936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c2f94f01-0623-4a8b-8fb0-309ca75d6725-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-sb57f\" (UID: \"c2f94f01-0623-4a8b-8fb0-309ca75d6725\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.752148 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.751981    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c2f94f01-0623-4a8b-8fb0-309ca75d6725-data\") pod \"seaweedfs-tls-custom-8b44d767d-sb57f\" (UID: \"c2f94f01-0623-4a8b-8fb0-309ca75d6725\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.752148 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.752021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znwqw\" (UniqueName: \"kubernetes.io/projected/c2f94f01-0623-4a8b-8fb0-309ca75d6725-kube-api-access-znwqw\") pod \"seaweedfs-tls-custom-8b44d767d-sb57f\" (UID: \"c2f94f01-0623-4a8b-8fb0-309ca75d6725\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.752385 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.752363    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c2f94f01-0623-4a8b-8fb0-309ca75d6725-data\") pod \"seaweedfs-tls-custom-8b44d767d-sb57f\" (UID: \"c2f94f01-0623-4a8b-8fb0-309ca75d6725\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.754542 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.754526    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/c2f94f01-0623-4a8b-8fb0-309ca75d6725-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-sb57f\" (UID: \"c2f94f01-0623-4a8b-8fb0-309ca75d6725\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.762515 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.762491    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znwqw\" (UniqueName: \"kubernetes.io/projected/c2f94f01-0623-4a8b-8fb0-309ca75d6725-kube-api-access-znwqw\") pod \"seaweedfs-tls-custom-8b44d767d-sb57f\" (UID: \"c2f94f01-0623-4a8b-8fb0-309ca75d6725\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:17.926836 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:17.926804    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f"
Sep 09 21:01:18.197287 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.197259    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-sb57f"]
Sep 09 21:01:18.200204 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:01:18.200175    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f94f01_0623_4a8b_8fb0_309ca75d6725.slice/crio-f3e368847bd1ef833f9c6b2cbd875d80c386a5e3a3a85346547dd09b1370293a WatchSource:0}: Error finding container f3e368847bd1ef833f9c6b2cbd875d80c386a5e3a3a85346547dd09b1370293a: Status 404 returned error can't find the container with id f3e368847bd1ef833f9c6b2cbd875d80c386a5e3a3a85346547dd09b1370293a
Sep 09 21:01:18.567265 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.567220    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f" event={"ID":"c2f94f01-0623-4a8b-8fb0-309ca75d6725","Type":"ContainerStarted","Data":"d853a49c53c77804f602f9d4de45d09b265847f9329bfb307921cb33ad5f2c84"}
Sep 09 21:01:18.567462 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.567321    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f" event={"ID":"c2f94f01-0623-4a8b-8fb0-309ca75d6725","Type":"ContainerStarted","Data":"f3e368847bd1ef833f9c6b2cbd875d80c386a5e3a3a85346547dd09b1370293a"}
Sep 09 21:01:18.585629 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.585572    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-sb57f" podStartSLOduration=1.3569019070000001 podStartE2EDuration="1.585556851s" totalImagesPullingTime="228.654944ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:18.584689468 +0000 UTC m=+568.145716786" watchObservedRunningTime="2026-09-09 21:01:18.585556851 +0000 UTC m=+568.146584191"
Sep 09 21:01:18.868884 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.868851    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-vh68l"]
Sep 09 21:01:18.871273 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.871258    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:18.873438 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.873421    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 21:01:18.880982 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.880960    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-vh68l"]
Sep 09 21:01:18.935336 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.935301    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7eb553b-3a59-4616-b8b5-6ff7950c95b1" path="/var/lib/kubelet/pods/e7eb553b-3a59-4616-b8b5-6ff7950c95b1/volumes"
Sep 09 21:01:18.962567 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.962530    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhwcc\" (UniqueName: \"kubernetes.io/projected/1d4dac54-92a9-483e-940a-cc609a72c94e-kube-api-access-bhwcc\") pod \"s3-tls-init-custom-vh68l\" (UID: \"1d4dac54-92a9-483e-940a-cc609a72c94e\") " pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:18.962722 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:18.962576    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4dac54-92a9-483e-940a-cc609a72c94e-model-cache\") pod \"s3-tls-init-custom-vh68l\" (UID: \"1d4dac54-92a9-483e-940a-cc609a72c94e\") " pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:19.063275 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:19.063230    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhwcc\" (UniqueName: \"kubernetes.io/projected/1d4dac54-92a9-483e-940a-cc609a72c94e-kube-api-access-bhwcc\") pod \"s3-tls-init-custom-vh68l\" (UID: \"1d4dac54-92a9-483e-940a-cc609a72c94e\") " pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:19.063275 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:19.063270    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4dac54-92a9-483e-940a-cc609a72c94e-model-cache\") pod \"s3-tls-init-custom-vh68l\" (UID: \"1d4dac54-92a9-483e-940a-cc609a72c94e\") " pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:19.063625 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:19.063610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4dac54-92a9-483e-940a-cc609a72c94e-model-cache\") pod \"s3-tls-init-custom-vh68l\" (UID: \"1d4dac54-92a9-483e-940a-cc609a72c94e\") " pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:19.072106 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:19.072075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhwcc\" (UniqueName: \"kubernetes.io/projected/1d4dac54-92a9-483e-940a-cc609a72c94e-kube-api-access-bhwcc\") pod \"s3-tls-init-custom-vh68l\" (UID: \"1d4dac54-92a9-483e-940a-cc609a72c94e\") " pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:19.188785 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:19.188698    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:19.460822 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:19.460793    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-vh68l"]
Sep 09 21:01:19.463396 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:01:19.463370    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d4dac54_92a9_483e_940a_cc609a72c94e.slice/crio-66263a623209778c9592141480ee3c1a5c7d7e6e64ada748edbc0139398cdcde WatchSource:0}: Error finding container 66263a623209778c9592141480ee3c1a5c7d7e6e64ada748edbc0139398cdcde: Status 404 returned error can't find the container with id 66263a623209778c9592141480ee3c1a5c7d7e6e64ada748edbc0139398cdcde
Sep 09 21:01:19.574199 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:19.574157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vh68l" event={"ID":"1d4dac54-92a9-483e-940a-cc609a72c94e","Type":"ContainerStarted","Data":"fd86374fdee385164a09a92b8df4af1068a40d9228322ca853326da061096b1d"}
Sep 09 21:01:19.574199 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:19.574204    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vh68l" event={"ID":"1d4dac54-92a9-483e-940a-cc609a72c94e","Type":"ContainerStarted","Data":"66263a623209778c9592141480ee3c1a5c7d7e6e64ada748edbc0139398cdcde"}
Sep 09 21:01:21.584476 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:21.584399    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d4dac54-92a9-483e-940a-cc609a72c94e" containerID="fd86374fdee385164a09a92b8df4af1068a40d9228322ca853326da061096b1d" exitCode=0
Sep 09 21:01:21.584476 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:21.584453    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vh68l" event={"ID":"1d4dac54-92a9-483e-940a-cc609a72c94e","Type":"ContainerDied","Data":"fd86374fdee385164a09a92b8df4af1068a40d9228322ca853326da061096b1d"}
Sep 09 21:01:22.591255 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:22.591221    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vh68l" event={"ID":"1d4dac54-92a9-483e-940a-cc609a72c94e","Type":"ContainerStarted","Data":"02f92c3a78cc8532f6b7f610891d50c4c1873825e9ff029d390209ae1e7aa11d"}
Sep 09 21:01:22.612607 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:22.612553    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-vh68l" podStartSLOduration=2.875014225 podStartE2EDuration="4.612536412s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.737522187s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:22.611158124 +0000 UTC m=+572.172185442" watchObservedRunningTime="2026-09-09 21:01:22.612536412 +0000 UTC m=+572.173563728"
Sep 09 21:01:28.623634 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:28.623548    2580 generic.go:356] "Generic (PLEG): container finished" podID="1d4dac54-92a9-483e-940a-cc609a72c94e" containerID="02f92c3a78cc8532f6b7f610891d50c4c1873825e9ff029d390209ae1e7aa11d" exitCode=0
Sep 09 21:01:28.623634 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:28.623615    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vh68l" event={"ID":"1d4dac54-92a9-483e-940a-cc609a72c94e","Type":"ContainerDied","Data":"02f92c3a78cc8532f6b7f610891d50c4c1873825e9ff029d390209ae1e7aa11d"}
Sep 09 21:01:29.767560 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:29.767534    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:29.849099 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:29.849060    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4dac54-92a9-483e-940a-cc609a72c94e-model-cache\") pod \"1d4dac54-92a9-483e-940a-cc609a72c94e\" (UID: \"1d4dac54-92a9-483e-940a-cc609a72c94e\") "
Sep 09 21:01:29.849299 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:29.849130    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhwcc\" (UniqueName: \"kubernetes.io/projected/1d4dac54-92a9-483e-940a-cc609a72c94e-kube-api-access-bhwcc\") pod \"1d4dac54-92a9-483e-940a-cc609a72c94e\" (UID: \"1d4dac54-92a9-483e-940a-cc609a72c94e\") "
Sep 09 21:01:29.851354 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:29.851323    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d4dac54-92a9-483e-940a-cc609a72c94e-kube-api-access-bhwcc" (OuterVolumeSpecName: "kube-api-access-bhwcc") pod "1d4dac54-92a9-483e-940a-cc609a72c94e" (UID: "1d4dac54-92a9-483e-940a-cc609a72c94e"). InnerVolumeSpecName "kube-api-access-bhwcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:29.874078 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:29.874048    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d4dac54-92a9-483e-940a-cc609a72c94e-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d4dac54-92a9-483e-940a-cc609a72c94e" (UID: "1d4dac54-92a9-483e-940a-cc609a72c94e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:29.950068 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:29.949978    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d4dac54-92a9-483e-940a-cc609a72c94e-model-cache\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:01:29.950068 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:29.950019    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhwcc\" (UniqueName: \"kubernetes.io/projected/1d4dac54-92a9-483e-940a-cc609a72c94e-kube-api-access-bhwcc\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:01:30.634413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:30.634383    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vh68l"
Sep 09 21:01:30.634413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:30.634397    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vh68l" event={"ID":"1d4dac54-92a9-483e-940a-cc609a72c94e","Type":"ContainerDied","Data":"66263a623209778c9592141480ee3c1a5c7d7e6e64ada748edbc0139398cdcde"}
Sep 09 21:01:30.634621 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:30.634436    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66263a623209778c9592141480ee3c1a5c7d7e6e64ada748edbc0139398cdcde"
Sep 09 21:01:31.308607 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.308573    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"]
Sep 09 21:01:31.308999 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.308946    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4dac54-92a9-483e-940a-cc609a72c94e" containerName="s3-tls-init-custom"
Sep 09 21:01:31.308999 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.308958    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4dac54-92a9-483e-940a-cc609a72c94e" containerName="s3-tls-init-custom"
Sep 09 21:01:31.308999 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.308970    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d4dac54-92a9-483e-940a-cc609a72c94e" containerName="download-hf-model"
Sep 09 21:01:31.308999 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.308975    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d4dac54-92a9-483e-940a-cc609a72c94e" containerName="download-hf-model"
Sep 09 21:01:31.309121 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.309023    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d4dac54-92a9-483e-940a-cc609a72c94e" containerName="s3-tls-init-custom"
Sep 09 21:01:31.311013 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.310998    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.314963 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.314943    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 09 21:01:31.315093 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.314973    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 09 21:01:31.325944 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.325922    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"]
Sep 09 21:01:31.363408 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.363372    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfg5w\" (UniqueName: \"kubernetes.io/projected/da71c3f6-3df0-4dcc-8203-6cefc4023785-kube-api-access-lfg5w\") pod \"seaweedfs-tls-serving-67b7477c6d-9rcb7\" (UID: \"da71c3f6-3df0-4dcc-8203-6cefc4023785\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.363560 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.363429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/da71c3f6-3df0-4dcc-8203-6cefc4023785-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-9rcb7\" (UID: \"da71c3f6-3df0-4dcc-8203-6cefc4023785\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.363560 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.363508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/da71c3f6-3df0-4dcc-8203-6cefc4023785-data\") pod \"seaweedfs-tls-serving-67b7477c6d-9rcb7\" (UID: \"da71c3f6-3df0-4dcc-8203-6cefc4023785\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.464033 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.463997    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/da71c3f6-3df0-4dcc-8203-6cefc4023785-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-9rcb7\" (UID: \"da71c3f6-3df0-4dcc-8203-6cefc4023785\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.464199 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.464061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/da71c3f6-3df0-4dcc-8203-6cefc4023785-data\") pod \"seaweedfs-tls-serving-67b7477c6d-9rcb7\" (UID: \"da71c3f6-3df0-4dcc-8203-6cefc4023785\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.464199 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.464102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfg5w\" (UniqueName: \"kubernetes.io/projected/da71c3f6-3df0-4dcc-8203-6cefc4023785-kube-api-access-lfg5w\") pod \"seaweedfs-tls-serving-67b7477c6d-9rcb7\" (UID: \"da71c3f6-3df0-4dcc-8203-6cefc4023785\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.464528 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.464502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/da71c3f6-3df0-4dcc-8203-6cefc4023785-data\") pod \"seaweedfs-tls-serving-67b7477c6d-9rcb7\" (UID: \"da71c3f6-3df0-4dcc-8203-6cefc4023785\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.466524 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.466507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/da71c3f6-3df0-4dcc-8203-6cefc4023785-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-9rcb7\" (UID: \"da71c3f6-3df0-4dcc-8203-6cefc4023785\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.473141 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.473119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfg5w\" (UniqueName: \"kubernetes.io/projected/da71c3f6-3df0-4dcc-8203-6cefc4023785-kube-api-access-lfg5w\") pod \"seaweedfs-tls-serving-67b7477c6d-9rcb7\" (UID: \"da71c3f6-3df0-4dcc-8203-6cefc4023785\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.619486 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.619458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"
Sep 09 21:01:31.891404 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:31.891369    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7"]
Sep 09 21:01:31.893620 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:01:31.893590    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda71c3f6_3df0_4dcc_8203_6cefc4023785.slice/crio-903c1663ee77960b6ee8249879f316d989b456ffed453ecab31739bd43f7a002 WatchSource:0}: Error finding container 903c1663ee77960b6ee8249879f316d989b456ffed453ecab31739bd43f7a002: Status 404 returned error can't find the container with id 903c1663ee77960b6ee8249879f316d989b456ffed453ecab31739bd43f7a002
Sep 09 21:01:32.645711 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:32.645674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7" event={"ID":"da71c3f6-3df0-4dcc-8203-6cefc4023785","Type":"ContainerStarted","Data":"9606844c7b65c38d959e8bd016f45be1cf40cf16253e68744efac939b39f573f"}
Sep 09 21:01:32.646153 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:32.645725    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7" event={"ID":"da71c3f6-3df0-4dcc-8203-6cefc4023785","Type":"ContainerStarted","Data":"903c1663ee77960b6ee8249879f316d989b456ffed453ecab31739bd43f7a002"}
Sep 09 21:01:32.663746 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:32.663695    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-9rcb7" podStartSLOduration=1.388781917 podStartE2EDuration="1.663678259s" totalImagesPullingTime="274.896342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:32.662632326 +0000 UTC m=+582.223659643" watchObservedRunningTime="2026-09-09 21:01:32.663678259 +0000 UTC m=+582.224705613"
Sep 09 21:01:33.226575 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.226541    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-hgdvj"]
Sep 09 21:01:33.229898 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.229879    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:33.232607 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.232588    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 09 21:01:33.239413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.239389    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-hgdvj"]
Sep 09 21:01:33.277914 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.277877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-model-cache\") pod \"s3-tls-init-serving-hgdvj\" (UID: \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\") " pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:33.277914 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.277921    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g87b\" (UniqueName: \"kubernetes.io/projected/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-kube-api-access-6g87b\") pod \"s3-tls-init-serving-hgdvj\" (UID: \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\") " pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:33.378508 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.378470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-model-cache\") pod \"s3-tls-init-serving-hgdvj\" (UID: \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\") " pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:33.378508 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.378510    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g87b\" (UniqueName: \"kubernetes.io/projected/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-kube-api-access-6g87b\") pod \"s3-tls-init-serving-hgdvj\" (UID: \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\") " pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:33.378925 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.378900    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-model-cache\") pod \"s3-tls-init-serving-hgdvj\" (UID: \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\") " pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:33.388144 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.388118    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g87b\" (UniqueName: \"kubernetes.io/projected/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-kube-api-access-6g87b\") pod \"s3-tls-init-serving-hgdvj\" (UID: \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\") " pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:33.553681 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.553596    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:33.831328 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:33.831256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-hgdvj"]
Sep 09 21:01:33.834216 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:01:33.834186    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a2cf039_eb5b_4f52_a0a6_aeaf0c91d6e2.slice/crio-c6ccf020ad262f1fda95f6e9c85101d43d0eed9fe7ba8df9fe65861d5b4b40e4 WatchSource:0}: Error finding container c6ccf020ad262f1fda95f6e9c85101d43d0eed9fe7ba8df9fe65861d5b4b40e4: Status 404 returned error can't find the container with id c6ccf020ad262f1fda95f6e9c85101d43d0eed9fe7ba8df9fe65861d5b4b40e4
Sep 09 21:01:34.658106 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:34.658072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgdvj" event={"ID":"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2","Type":"ContainerStarted","Data":"c775b9ea76c50a358b12158079323fd6b4245adf31840a1eb539908364f922e6"}
Sep 09 21:01:34.658279 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:34.658114    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgdvj" event={"ID":"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2","Type":"ContainerStarted","Data":"c6ccf020ad262f1fda95f6e9c85101d43d0eed9fe7ba8df9fe65861d5b4b40e4"}
Sep 09 21:01:36.668532 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:36.668502    2580 generic.go:356] "Generic (PLEG): container finished" podID="0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2" containerID="c775b9ea76c50a358b12158079323fd6b4245adf31840a1eb539908364f922e6" exitCode=0
Sep 09 21:01:36.668912 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:36.668551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgdvj" event={"ID":"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2","Type":"ContainerDied","Data":"c775b9ea76c50a358b12158079323fd6b4245adf31840a1eb539908364f922e6"}
Sep 09 21:01:37.678633 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:37.678597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgdvj" event={"ID":"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2","Type":"ContainerStarted","Data":"69dc57c40cef84de104e2407aa19171d25f81ca2461fe54da73b93dd1e278a0d"}
Sep 09 21:01:37.701527 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:37.701477    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-hgdvj" podStartSLOduration=2.8932719430000002 podStartE2EDuration="4.70146139s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.808189447s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:37.699671226 +0000 UTC m=+587.260698534" watchObservedRunningTime="2026-09-09 21:01:37.70146139 +0000 UTC m=+587.262488707"
Sep 09 21:01:43.711713 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:43.711642    2580 generic.go:356] "Generic (PLEG): container finished" podID="0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2" containerID="69dc57c40cef84de104e2407aa19171d25f81ca2461fe54da73b93dd1e278a0d" exitCode=0
Sep 09 21:01:43.712153 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:43.711711    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgdvj" event={"ID":"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2","Type":"ContainerDied","Data":"69dc57c40cef84de104e2407aa19171d25f81ca2461fe54da73b93dd1e278a0d"}
Sep 09 21:01:44.856496 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:44.856474    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:44.985923 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:44.985849    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-model-cache\") pod \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\" (UID: \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\") "
Sep 09 21:01:44.985923 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:44.985884    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g87b\" (UniqueName: \"kubernetes.io/projected/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-kube-api-access-6g87b\") pod \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\" (UID: \"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2\") "
Sep 09 21:01:44.988230 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:44.988199    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-kube-api-access-6g87b" (OuterVolumeSpecName: "kube-api-access-6g87b") pod "0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2" (UID: "0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2"). InnerVolumeSpecName "kube-api-access-6g87b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:45.007418 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:45.007389    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2" (UID: "0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:45.087275 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:45.087250    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-model-cache\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:01:45.087275 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:45.087271    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g87b\" (UniqueName: \"kubernetes.io/projected/0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2-kube-api-access-6g87b\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:01:45.722627 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:45.722597    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hgdvj"
Sep 09 21:01:45.722836 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:45.722602    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hgdvj" event={"ID":"0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2","Type":"ContainerDied","Data":"c6ccf020ad262f1fda95f6e9c85101d43d0eed9fe7ba8df9fe65861d5b4b40e4"}
Sep 09 21:01:45.722836 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:45.722705    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6ccf020ad262f1fda95f6e9c85101d43d0eed9fe7ba8df9fe65861d5b4b40e4"
Sep 09 21:01:50.866152 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:50.866125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:01:50.868321 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:50.868304    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:01:58.705294 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.705255    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"]
Sep 09 21:01:58.706739 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.705639    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2" containerName="download-hf-model"
Sep 09 21:01:58.706739 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.705653    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2" containerName="download-hf-model"
Sep 09 21:01:58.706739 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.705667    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2" containerName="s3-tls-init-serving"
Sep 09 21:01:58.706739 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.705673    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2" containerName="s3-tls-init-serving"
Sep 09 21:01:58.706739 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.705731    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2" containerName="s3-tls-init-serving"
Sep 09 21:01:58.707851 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.707835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.714174 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.714153    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-42176a\""
Sep 09 21:01:58.714174 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.714164    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:01:58.714379 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.714167    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-42176a-predictor-serving-cert\""
Sep 09 21:01:58.714938 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.714915    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-42176a-dockercfg-5tptb\""
Sep 09 21:01:58.715069 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.714964    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-42176a-kube-rbac-proxy-sar-config\""
Sep 09 21:01:58.715069 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.715020    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:01:58.715069 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.715023    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:01:58.736219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.736194    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"]
Sep 09 21:01:58.802637 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.802600    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vn5\" (UniqueName: \"kubernetes.io/projected/d7731184-110a-47d8-809d-057af2f26558-kube-api-access-z8vn5\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.802824 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.802712    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.802824 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.802751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-cabundle-cert\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.802824 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.802821    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7731184-110a-47d8-809d-057af2f26558-kserve-provision-location\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.802965 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.802893    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7731184-110a-47d8-809d-057af2f26558-proxy-tls\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.903577 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.903542    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.903785 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.903580    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-cabundle-cert\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.903785 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.903609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7731184-110a-47d8-809d-057af2f26558-kserve-provision-location\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.903785 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.903648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7731184-110a-47d8-809d-057af2f26558-proxy-tls\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.903785 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.903675    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vn5\" (UniqueName: \"kubernetes.io/projected/d7731184-110a-47d8-809d-057af2f26558-kube-api-access-z8vn5\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.904142 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.904120    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7731184-110a-47d8-809d-057af2f26558-kserve-provision-location\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.904352 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.904331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-cabundle-cert\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.904436 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.904338    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.906323 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.906306    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7731184-110a-47d8-809d-057af2f26558-proxy-tls\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.915515 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.915490    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vn5\" (UniqueName: \"kubernetes.io/projected/d7731184-110a-47d8-809d-057af2f26558-kube-api-access-z8vn5\") pod \"isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") " pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:58.941700 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.941677    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"]
Sep 09 21:01:58.944994 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.944980    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:58.948488 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.948469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 09 21:01:58.948812 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.948796    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:01:58.950467 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.950449    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:01:58.950551 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.950451    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 09 21:01:58.997787 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:58.997695    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"]
Sep 09 21:01:59.018512 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.018485    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:01:59.105397 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.105366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8e3a316-4557-4819-98ef-489e732e3cf7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.105545 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.105413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fjs9\" (UniqueName: \"kubernetes.io/projected/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-api-access-8fjs9\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.105545 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.105487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8e3a316-4557-4819-98ef-489e732e3cf7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.105545 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.105524    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.105668 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.105610    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.206063 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.206026    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.206063 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.206068    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8e3a316-4557-4819-98ef-489e732e3cf7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.206301 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.206091    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fjs9\" (UniqueName: \"kubernetes.io/projected/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-api-access-8fjs9\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.206301 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.206130    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8e3a316-4557-4819-98ef-489e732e3cf7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.206301 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.206166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.206603 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.206575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8e3a316-4557-4819-98ef-489e732e3cf7-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.206790 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.206747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.206790 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.206780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.208740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.208716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8e3a316-4557-4819-98ef-489e732e3cf7-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.219330 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.219310    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fjs9\" (UniqueName: \"kubernetes.io/projected/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-api-access-8fjs9\") pod \"isvc-sklearn-batcher-predictor-7fbc757658-78b4f\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.257126 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.257078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:01:59.301786 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.301746    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"]
Sep 09 21:01:59.302988 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:01:59.302965    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7731184_110a_47d8_809d_057af2f26558.slice/crio-589e0ce4153461ca40250a5449c12b8b61d645a02601839d74ff29bf570f75fc WatchSource:0}: Error finding container 589e0ce4153461ca40250a5449c12b8b61d645a02601839d74ff29bf570f75fc: Status 404 returned error can't find the container with id 589e0ce4153461ca40250a5449c12b8b61d645a02601839d74ff29bf570f75fc
Sep 09 21:01:59.540152 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.540123    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"]
Sep 09 21:01:59.542268 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:01:59.542238    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e3a316_4557_4819_98ef_489e732e3cf7.slice/crio-13969f4045e19b3ef92aae3051dc47762823987ce24b9811843f858654c4c71b WatchSource:0}: Error finding container 13969f4045e19b3ef92aae3051dc47762823987ce24b9811843f858654c4c71b: Status 404 returned error can't find the container with id 13969f4045e19b3ef92aae3051dc47762823987ce24b9811843f858654c4c71b
Sep 09 21:01:59.793917 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.793815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2" event={"ID":"d7731184-110a-47d8-809d-057af2f26558","Type":"ContainerStarted","Data":"900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3"}
Sep 09 21:01:59.793917 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.793897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2" event={"ID":"d7731184-110a-47d8-809d-057af2f26558","Type":"ContainerStarted","Data":"589e0ce4153461ca40250a5449c12b8b61d645a02601839d74ff29bf570f75fc"}
Sep 09 21:01:59.795881 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.795847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerStarted","Data":"d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502"}
Sep 09 21:01:59.795986 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:01:59.795924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerStarted","Data":"13969f4045e19b3ef92aae3051dc47762823987ce24b9811843f858654c4c71b"}
Sep 09 21:02:03.818365 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:03.818277    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerID="d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502" exitCode=0
Sep 09 21:02:03.818365 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:03.818354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerDied","Data":"d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502"}
Sep 09 21:02:06.842011 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:06.841978    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2_d7731184-110a-47d8-809d-057af2f26558/storage-initializer/0.log"
Sep 09 21:02:06.842550 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:06.842027    2580 generic.go:356] "Generic (PLEG): container finished" podID="d7731184-110a-47d8-809d-057af2f26558" containerID="900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3" exitCode=1
Sep 09 21:02:06.842550 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:06.842153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2" event={"ID":"d7731184-110a-47d8-809d-057af2f26558","Type":"ContainerDied","Data":"900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3"}
Sep 09 21:02:07.883042 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:07.883003    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2_d7731184-110a-47d8-809d-057af2f26558/storage-initializer/0.log"
Sep 09 21:02:07.883499 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:07.883139    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2" event={"ID":"d7731184-110a-47d8-809d-057af2f26558","Type":"ContainerStarted","Data":"a9b09bc185da3f8ffd274198047a3679cab3abd4bfa4ace3d7cfad8d3b1c9c71"}
Sep 09 21:02:08.494538 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:08.494502    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"]
Sep 09 21:02:08.892540 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:08.892475    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2" podUID="d7731184-110a-47d8-809d-057af2f26558" containerName="storage-initializer" containerID="cri-o://a9b09bc185da3f8ffd274198047a3679cab3abd4bfa4ace3d7cfad8d3b1c9c71" gracePeriod=30
Sep 09 21:02:10.909802 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:10.909701    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2_d7731184-110a-47d8-809d-057af2f26558/storage-initializer/1.log"
Sep 09 21:02:10.910305 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:10.910287    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2_d7731184-110a-47d8-809d-057af2f26558/storage-initializer/0.log"
Sep 09 21:02:10.910407 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:10.910329    2580 generic.go:356] "Generic (PLEG): container finished" podID="d7731184-110a-47d8-809d-057af2f26558" containerID="a9b09bc185da3f8ffd274198047a3679cab3abd4bfa4ace3d7cfad8d3b1c9c71" exitCode=1
Sep 09 21:02:10.910465 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:10.910409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2" event={"ID":"d7731184-110a-47d8-809d-057af2f26558","Type":"ContainerDied","Data":"a9b09bc185da3f8ffd274198047a3679cab3abd4bfa4ace3d7cfad8d3b1c9c71"}
Sep 09 21:02:10.910533 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:10.910518    2580 scope.go:117] "RemoveContainer" containerID="900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3"
Sep 09 21:02:11.026919 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.026893    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2_d7731184-110a-47d8-809d-057af2f26558/storage-initializer/1.log"
Sep 09 21:02:11.028634 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.028615    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:02:11.116987 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.116955    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-kube-rbac-proxy-sar-config\") pod \"d7731184-110a-47d8-809d-057af2f26558\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") "
Sep 09 21:02:11.117186 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.117007    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7731184-110a-47d8-809d-057af2f26558-proxy-tls\") pod \"d7731184-110a-47d8-809d-057af2f26558\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") "
Sep 09 21:02:11.117186 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.117058    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8vn5\" (UniqueName: \"kubernetes.io/projected/d7731184-110a-47d8-809d-057af2f26558-kube-api-access-z8vn5\") pod \"d7731184-110a-47d8-809d-057af2f26558\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") "
Sep 09 21:02:11.117186 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.117088    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-cabundle-cert\") pod \"d7731184-110a-47d8-809d-057af2f26558\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") "
Sep 09 21:02:11.117457 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.117418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7731184-110a-47d8-809d-057af2f26558" (UID: "d7731184-110a-47d8-809d-057af2f26558"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:11.117600 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.117509    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d7731184-110a-47d8-809d-057af2f26558" (UID: "d7731184-110a-47d8-809d-057af2f26558"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:02:11.119540 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.119514    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7731184-110a-47d8-809d-057af2f26558-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7731184-110a-47d8-809d-057af2f26558" (UID: "d7731184-110a-47d8-809d-057af2f26558"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:02:11.119655 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.119621    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7731184-110a-47d8-809d-057af2f26558-kube-api-access-z8vn5" (OuterVolumeSpecName: "kube-api-access-z8vn5") pod "d7731184-110a-47d8-809d-057af2f26558" (UID: "d7731184-110a-47d8-809d-057af2f26558"). InnerVolumeSpecName "kube-api-access-z8vn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:02:11.218119 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.218023    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7731184-110a-47d8-809d-057af2f26558-kserve-provision-location\") pod \"d7731184-110a-47d8-809d-057af2f26558\" (UID: \"d7731184-110a-47d8-809d-057af2f26558\") "
Sep 09 21:02:11.218272 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.218226    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8vn5\" (UniqueName: \"kubernetes.io/projected/d7731184-110a-47d8-809d-057af2f26558-kube-api-access-z8vn5\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:02:11.218272 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.218240    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:02:11.218272 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.218253    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7731184-110a-47d8-809d-057af2f26558-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:02:11.218272 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.218267    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7731184-110a-47d8-809d-057af2f26558-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:02:11.218436 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.218306    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7731184-110a-47d8-809d-057af2f26558-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7731184-110a-47d8-809d-057af2f26558" (UID: "d7731184-110a-47d8-809d-057af2f26558"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:02:11.319224 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.319192    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7731184-110a-47d8-809d-057af2f26558-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:02:11.918181 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.918154    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2_d7731184-110a-47d8-809d-057af2f26558/storage-initializer/1.log"
Sep 09 21:02:11.918809 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.918784    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"
Sep 09 21:02:11.918966 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.918780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2" event={"ID":"d7731184-110a-47d8-809d-057af2f26558","Type":"ContainerDied","Data":"589e0ce4153461ca40250a5449c12b8b61d645a02601839d74ff29bf570f75fc"}
Sep 09 21:02:11.962208 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.962123    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"]
Sep 09 21:02:11.968528 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:11.968498    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-42176a-predictor-56d55d6cd4-q6vr2"]
Sep 09 21:02:12.936446 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:12.936410    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7731184-110a-47d8-809d-057af2f26558" path="/var/lib/kubelet/pods/d7731184-110a-47d8-809d-057af2f26558/volumes"
Sep 09 21:02:13.694664 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.694621    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"]
Sep 09 21:02:13.695210 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.695191    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7731184-110a-47d8-809d-057af2f26558" containerName="storage-initializer"
Sep 09 21:02:13.695210 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.695211    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7731184-110a-47d8-809d-057af2f26558" containerName="storage-initializer"
Sep 09 21:02:13.695335 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.695294    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7731184-110a-47d8-809d-057af2f26558" containerName="storage-initializer"
Sep 09 21:02:13.695335 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.695305    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7731184-110a-47d8-809d-057af2f26558" containerName="storage-initializer"
Sep 09 21:02:13.695424 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.695411    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7731184-110a-47d8-809d-057af2f26558" containerName="storage-initializer"
Sep 09 21:02:13.695479 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.695425    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7731184-110a-47d8-809d-057af2f26558" containerName="storage-initializer"
Sep 09 21:02:13.698143 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.698125    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.700249 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.700216    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 09 21:02:13.700249 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.700233    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 09 21:02:13.710087 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.710068    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"]
Sep 09 21:02:13.734977 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.734939    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.735126 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.735019    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.735126 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.735090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.735235 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.735133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.735235 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.735185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69dnj\" (UniqueName: \"kubernetes.io/projected/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-api-access-69dnj\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.836477 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.836440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.836658 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.836507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69dnj\" (UniqueName: \"kubernetes.io/projected/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-api-access-69dnj\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.836658 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.836549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.836658 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.836586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.836658 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.836642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.837605 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.837031    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.837605 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.837529    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.837605 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.837564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.839531 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.839511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:13.846324 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:13.846276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69dnj\" (UniqueName: \"kubernetes.io/projected/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-api-access-69dnj\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-fmhl8\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:14.010714 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:14.010643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:14.515197 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:14.515172    2580 scope.go:117] "RemoveContainer" containerID="a9b09bc185da3f8ffd274198047a3679cab3abd4bfa4ace3d7cfad8d3b1c9c71"
Sep 09 21:02:19.616531 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:19.616280    2580 scope.go:117] "RemoveContainer" containerID="900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3"
Sep 09 21:02:19.616848 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:02:19.616686    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3\": container with ID starting with 900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3 not found: ID does not exist" containerID="900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3"
Sep 09 21:02:19.616848 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:19.616718    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3"} err="failed to get container status \"900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3\": rpc error: code = NotFound desc = could not find container \"900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3\": container with ID starting with 900ad36aa5ecd46d09c8f98297939a761ce6ce706ee1069e172ea859a76aabe3 not found: ID does not exist"
Sep 09 21:02:19.893248 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:19.889859    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"]
Sep 09 21:02:20.071816 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:02:20.071744    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1347a1_0ad3_41a9_8dd1_c57bbe692d4e.slice/crio-49b59daf77faa39ce1de41a32856e22b3fa18feb036ddcad864611a505ab32be WatchSource:0}: Error finding container 49b59daf77faa39ce1de41a32856e22b3fa18feb036ddcad864611a505ab32be: Status 404 returned error can't find the container with id 49b59daf77faa39ce1de41a32856e22b3fa18feb036ddcad864611a505ab32be
Sep 09 21:02:20.976499 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:20.976462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerStarted","Data":"44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940"}
Sep 09 21:02:20.978417 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:20.978384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" event={"ID":"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e","Type":"ContainerStarted","Data":"493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7"}
Sep 09 21:02:20.978561 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:20.978450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" event={"ID":"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e","Type":"ContainerStarted","Data":"49b59daf77faa39ce1de41a32856e22b3fa18feb036ddcad864611a505ab32be"}
Sep 09 21:02:22.991190 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:22.991158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerStarted","Data":"f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e"}
Sep 09 21:02:23.998117 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:23.998078    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerID="493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7" exitCode=0
Sep 09 21:02:23.998563 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:23.998140    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" event={"ID":"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e","Type":"ContainerDied","Data":"493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7"}
Sep 09 21:02:27.031849 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:27.031809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerStarted","Data":"3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512"}
Sep 09 21:02:27.032396 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:27.032367    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:02:27.032513 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:27.032436    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:02:27.032670 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:27.032575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:02:27.035916 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:27.035686    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:02:27.036500 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:27.036426    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:27.043231 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:27.043211    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:02:27.054246 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:27.053697    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podStartSLOduration=2.646342906 podStartE2EDuration="29.053678394s" totalImagesPullingTime="22.563453353s" totalInitContainerRuntime="3.843882135s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:27.052677453 +0000 UTC m=+636.613704771" watchObservedRunningTime="2026-09-09 21:02:27.053678394 +0000 UTC m=+636.614705712"
Sep 09 21:02:28.040659 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:28.040621    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:02:28.041236 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:28.041209    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:29.046805 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:29.046746    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:02:29.047410 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:29.047388    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:39.046954 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:39.046879    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:02:39.047466 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:39.047337    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:49.047219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:49.047177    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:02:49.047721 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:49.047689    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:02:49.167975 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:49.167933    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" event={"ID":"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e","Type":"ContainerStarted","Data":"f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6"}
Sep 09 21:02:49.168152 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:49.167997    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" event={"ID":"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e","Type":"ContainerStarted","Data":"c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963"}
Sep 09 21:02:49.168334 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:49.168313    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:49.197135 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:49.197081    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podStartSLOduration=8.132837945 podStartE2EDuration="36.197067876s" totalImagesPullingTime="24.400668529s" totalInitContainerRuntime="3.663561402s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:49.196795253 +0000 UTC m=+658.757822571" watchObservedRunningTime="2026-09-09 21:02:49.197067876 +0000 UTC m=+658.758095193"
Sep 09 21:02:50.173613 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:50.173581    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:50.175105 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:50.175078    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:02:51.178460 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:51.178372    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:02:56.183533 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:56.183505    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:02:56.184086 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:56.184053    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:02:59.047308 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:59.047260    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:02:59.047687 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:02:59.047659    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:06.184493 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:06.184446    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:03:09.047848 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:09.047801    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:03:09.048324 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:09.048292    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:16.184501 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:16.184459    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:03:19.047617 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:19.047574    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:03:19.048122 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:19.048098    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:26.184721 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:26.184683    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:03:29.047110 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:29.047059    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:03:29.047599 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:29.047548    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:30.933206 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:30.933160    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:03:30.933972 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:30.933944    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:03:36.184438 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:36.184394    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:03:40.936821 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:40.936790    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:03:40.937176 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:40.936873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:03:46.184088 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:46.184042    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:03:53.699309 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.699270    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"]
Sep 09 21:03:53.699713 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.699522    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" containerID="cri-o://44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940" gracePeriod=30
Sep 09 21:03:53.699713 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.699542    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" containerID="cri-o://3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512" gracePeriod=30
Sep 09 21:03:53.699713 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.699592    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy" containerID="cri-o://f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e" gracePeriod=30
Sep 09 21:03:53.763549 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.763514    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"]
Sep 09 21:03:53.767134 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.767114    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:53.769545 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.769522    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 09 21:03:53.769657 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.769572    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 09 21:03:53.779038 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.779014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"]
Sep 09 21:03:53.934938 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.934907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b1d1e7-d848-4405-9740-e92e3331eff5-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:53.934938 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.934940    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:53.935181 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.935151    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29b1d1e7-d848-4405-9740-e92e3331eff5-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:53.935181 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.935169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:53.935300 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:53.935250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gx9m\" (UniqueName: \"kubernetes.io/projected/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-api-access-6gx9m\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.035889 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.035796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29b1d1e7-d848-4405-9740-e92e3331eff5-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.035889 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.035832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.035889 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.035883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gx9m\" (UniqueName: \"kubernetes.io/projected/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-api-access-6gx9m\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.036178 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.035910    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b1d1e7-d848-4405-9740-e92e3331eff5-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.036178 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.035934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.036392 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.036365    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b1d1e7-d848-4405-9740-e92e3331eff5-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.036606 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.036561    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.036606 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.036623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.038633 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.038609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29b1d1e7-d848-4405-9740-e92e3331eff5-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.045230 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.045202    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gx9m\" (UniqueName: \"kubernetes.io/projected/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-api-access-6gx9m\") pod \"isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.077914 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.077882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:54.361920 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.361894    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"]
Sep 09 21:03:54.364049 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:03:54.364023    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b1d1e7_d848_4405_9740_e92e3331eff5.slice/crio-13c0213112e66c3498173a592cc0d94a6bc968221335b8a493f5b53e41da5c1f WatchSource:0}: Error finding container 13c0213112e66c3498173a592cc0d94a6bc968221335b8a493f5b53e41da5c1f: Status 404 returned error can't find the container with id 13c0213112e66c3498173a592cc0d94a6bc968221335b8a493f5b53e41da5c1f
Sep 09 21:03:54.490164 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.490121    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerStarted","Data":"d37fef4414006bec679dd6684438eb1ea818598aa8d9df45308b4abe9249066d"}
Sep 09 21:03:54.490358 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.490194    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerStarted","Data":"13c0213112e66c3498173a592cc0d94a6bc968221335b8a493f5b53e41da5c1f"}
Sep 09 21:03:54.492773 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.492727    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerID="f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e" exitCode=2
Sep 09 21:03:54.492896 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:54.492791    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerDied","Data":"f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e"}
Sep 09 21:03:56.184560 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:56.184517    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:03:57.033415 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:57.033378    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 21:03:58.513742 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:58.513660    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerID="44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940" exitCode=0
Sep 09 21:03:58.514142 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:58.513737    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerDied","Data":"44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940"}
Sep 09 21:03:58.515310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:58.515291    2580 generic.go:356] "Generic (PLEG): container finished" podID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerID="d37fef4414006bec679dd6684438eb1ea818598aa8d9df45308b4abe9249066d" exitCode=0
Sep 09 21:03:58.515414 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:58.515333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerDied","Data":"d37fef4414006bec679dd6684438eb1ea818598aa8d9df45308b4abe9249066d"}
Sep 09 21:03:59.522552 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:59.522519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerStarted","Data":"774fa2fed42d437475d9c07f7ab84d83c3bfa08765ba320df8577c08c2e17fb1"}
Sep 09 21:03:59.522943 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:59.522586    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerStarted","Data":"3b481d37125fa6200ef9edeff59cfdd9119599431975a062a4651970f39cc390"}
Sep 09 21:03:59.522943 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:59.522610    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerStarted","Data":"a8a2d043792a047aad56e40d7a8b4ac9a39b66bf461793681252d41cbd850b98"}
Sep 09 21:03:59.523024 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:59.522948    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:59.523024 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:59.522986    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:03:59.524611 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:59.524589    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:03:59.566513 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:03:59.566466    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podStartSLOduration=2.783442559 podStartE2EDuration="6.56644921s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.783006651s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:59.564887995 +0000 UTC m=+729.125915312" watchObservedRunningTime="2026-09-09 21:03:59.56644921 +0000 UTC m=+729.127476525"
Sep 09 21:04:00.528391 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:00.528357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:04:00.528867 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:00.528488    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:04:00.529560 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:00.529536    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:00.933578 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:00.933531    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:04:00.933929 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:00.933904    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:01.535568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:01.535531    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:04:01.536015 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:01.535941    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:02.033646 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:02.033605    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 21:04:06.184935 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:06.184905    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:04:06.541445 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:06.541372    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:04:06.542011 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:06.541982    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:04:06.542336 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:06.542311    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:07.033580 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:07.033538    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 21:04:07.033842 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:07.033671    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:04:10.933722 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:10.933679    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:04:10.934259 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:10.934174    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:12.033256 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:12.033211    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 21:04:13.790502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:13.790468    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"]
Sep 09 21:04:13.790918 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:13.790732    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" containerID="cri-o://c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963" gracePeriod=30
Sep 09 21:04:13.790918 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:13.790796    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kube-rbac-proxy" containerID="cri-o://f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6" gracePeriod=30
Sep 09 21:04:14.606716 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:14.606685    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerID="f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6" exitCode=2
Sep 09 21:04:14.606919 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:14.606745    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" event={"ID":"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e","Type":"ContainerDied","Data":"f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6"}
Sep 09 21:04:16.179234 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:16.179188    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 09 21:04:16.184157 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:16.184131    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 09 21:04:16.542593 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:16.542486    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:04:16.542948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:16.542922    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:17.032985 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:17.032937    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 21:04:19.042974 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.042948    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:04:19.142952 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.142866    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-rbac-proxy-sar-config\") pod \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") "
Sep 09 21:04:19.142952 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.142915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kserve-provision-location\") pod \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") "
Sep 09 21:04:19.143180 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.142956    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-proxy-tls\") pod \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") "
Sep 09 21:04:19.143180 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.143038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69dnj\" (UniqueName: \"kubernetes.io/projected/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-api-access-69dnj\") pod \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") "
Sep 09 21:04:19.143180 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.143069    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-cabundle-cert\") pod \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\" (UID: \"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e\") "
Sep 09 21:04:19.143324 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.143213    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" (UID: "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:19.143324 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.143260    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" (UID: "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:19.143467 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.143442    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" (UID: "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:19.143509 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.143459    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:19.143509 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.143497    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:19.145142 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.145118    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-api-access-69dnj" (OuterVolumeSpecName: "kube-api-access-69dnj") pod "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" (UID: "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e"). InnerVolumeSpecName "kube-api-access-69dnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:19.145255 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.145197    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" (UID: "9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:19.244614 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.244580    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:19.244614 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.244607    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69dnj\" (UniqueName: \"kubernetes.io/projected/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-kube-api-access-69dnj\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:19.244614 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.244616    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:19.634610 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.634577    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerID="c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963" exitCode=0
Sep 09 21:04:19.634802 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.634636    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"
Sep 09 21:04:19.634802 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.634631    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" event={"ID":"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e","Type":"ContainerDied","Data":"c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963"}
Sep 09 21:04:19.634802 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.634782    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8" event={"ID":"9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e","Type":"ContainerDied","Data":"49b59daf77faa39ce1de41a32856e22b3fa18feb036ddcad864611a505ab32be"}
Sep 09 21:04:19.634929 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.634823    2580 scope.go:117] "RemoveContainer" containerID="f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6"
Sep 09 21:04:19.643855 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.643833    2580 scope.go:117] "RemoveContainer" containerID="c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963"
Sep 09 21:04:19.651671 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.651656    2580 scope.go:117] "RemoveContainer" containerID="493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7"
Sep 09 21:04:19.657278 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.657251    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"]
Sep 09 21:04:19.660877 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.660829    2580 scope.go:117] "RemoveContainer" containerID="f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6"
Sep 09 21:04:19.661139 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:04:19.661119    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6\": container with ID starting with f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6 not found: ID does not exist" containerID="f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6"
Sep 09 21:04:19.661232 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.661150    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6"} err="failed to get container status \"f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6\": rpc error: code = NotFound desc = could not find container \"f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6\": container with ID starting with f36708838a866abe5225a73b1383e3d50e17708b53d2f20666cfb68b15a14cc6 not found: ID does not exist"
Sep 09 21:04:19.661232 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.661174    2580 scope.go:117] "RemoveContainer" containerID="c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963"
Sep 09 21:04:19.661485 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:04:19.661468    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963\": container with ID starting with c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963 not found: ID does not exist" containerID="c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963"
Sep 09 21:04:19.661528 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.661492    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963"} err="failed to get container status \"c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963\": rpc error: code = NotFound desc = could not find container \"c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963\": container with ID starting with c7aa3359c3fcf19660c96aa4818387ceaf95bdbe2a66fee696af634b9b285963 not found: ID does not exist"
Sep 09 21:04:19.661528 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.661510    2580 scope.go:117] "RemoveContainer" containerID="493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7"
Sep 09 21:04:19.661603 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.661588    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-fmhl8"]
Sep 09 21:04:19.661745 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:04:19.661731    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7\": container with ID starting with 493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7 not found: ID does not exist" containerID="493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7"
Sep 09 21:04:19.661821 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:19.661749    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7"} err="failed to get container status \"493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7\": rpc error: code = NotFound desc = could not find container \"493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7\": container with ID starting with 493e856b1a70c96cb3814c65589a40a13d3256f816957dd09a57110f1188a0f7 not found: ID does not exist"
Sep 09 21:04:20.933279 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:20.933239    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Sep 09 21:04:20.933729 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:20.933498    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:20.936687 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:20.936664    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" path="/var/lib/kubelet/pods/9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e/volumes"
Sep 09 21:04:20.937064 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:20.937052    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:04:20.937134 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:20.937123    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:04:22.033371 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:22.033326    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 09 21:04:23.845535 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.845512    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:04:23.982308 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982222    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8e3a316-4557-4819-98ef-489e732e3cf7-kserve-provision-location\") pod \"c8e3a316-4557-4819-98ef-489e732e3cf7\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") "
Sep 09 21:04:23.982308 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982263    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fjs9\" (UniqueName: \"kubernetes.io/projected/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-api-access-8fjs9\") pod \"c8e3a316-4557-4819-98ef-489e732e3cf7\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") "
Sep 09 21:04:23.982308 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982294    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-cabundle-cert\") pod \"c8e3a316-4557-4819-98ef-489e732e3cf7\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") "
Sep 09 21:04:23.982541 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982398    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8e3a316-4557-4819-98ef-489e732e3cf7-proxy-tls\") pod \"c8e3a316-4557-4819-98ef-489e732e3cf7\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") "
Sep 09 21:04:23.982541 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982436    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-rbac-proxy-sar-config\") pod \"c8e3a316-4557-4819-98ef-489e732e3cf7\" (UID: \"c8e3a316-4557-4819-98ef-489e732e3cf7\") "
Sep 09 21:04:23.982645 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982573    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8e3a316-4557-4819-98ef-489e732e3cf7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c8e3a316-4557-4819-98ef-489e732e3cf7" (UID: "c8e3a316-4557-4819-98ef-489e732e3cf7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:23.982645 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c8e3a316-4557-4819-98ef-489e732e3cf7" (UID: "c8e3a316-4557-4819-98ef-489e732e3cf7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:23.982907 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982884    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:23.983032 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982911    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c8e3a316-4557-4819-98ef-489e732e3cf7-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:23.983032 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.982920    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c8e3a316-4557-4819-98ef-489e732e3cf7" (UID: "c8e3a316-4557-4819-98ef-489e732e3cf7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:23.984609 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.984583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-api-access-8fjs9" (OuterVolumeSpecName: "kube-api-access-8fjs9") pod "c8e3a316-4557-4819-98ef-489e732e3cf7" (UID: "c8e3a316-4557-4819-98ef-489e732e3cf7"). InnerVolumeSpecName "kube-api-access-8fjs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:23.984609 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:23.984601    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8e3a316-4557-4819-98ef-489e732e3cf7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c8e3a316-4557-4819-98ef-489e732e3cf7" (UID: "c8e3a316-4557-4819-98ef-489e732e3cf7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:24.084035 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.084000    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fjs9\" (UniqueName: \"kubernetes.io/projected/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-api-access-8fjs9\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:24.084035 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.084035    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c8e3a316-4557-4819-98ef-489e732e3cf7-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:24.084250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.084051    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c8e3a316-4557-4819-98ef-489e732e3cf7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:04:24.663351 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.663317    2580 generic.go:356] "Generic (PLEG): container finished" podID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerID="3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512" exitCode=0
Sep 09 21:04:24.663548 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.663377    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"
Sep 09 21:04:24.663548 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.663406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerDied","Data":"3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512"}
Sep 09 21:04:24.663548 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.663473    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f" event={"ID":"c8e3a316-4557-4819-98ef-489e732e3cf7","Type":"ContainerDied","Data":"13969f4045e19b3ef92aae3051dc47762823987ce24b9811843f858654c4c71b"}
Sep 09 21:04:24.663548 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.663507    2580 scope.go:117] "RemoveContainer" containerID="3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512"
Sep 09 21:04:24.672653 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.672639    2580 scope.go:117] "RemoveContainer" containerID="f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e"
Sep 09 21:04:24.680499 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.680484    2580 scope.go:117] "RemoveContainer" containerID="44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940"
Sep 09 21:04:24.686458 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.686436    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"]
Sep 09 21:04:24.689106 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.689083    2580 scope.go:117] "RemoveContainer" containerID="d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502"
Sep 09 21:04:24.689279 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.689259    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-7fbc757658-78b4f"]
Sep 09 21:04:24.696549 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.696532    2580 scope.go:117] "RemoveContainer" containerID="3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512"
Sep 09 21:04:24.696823 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:04:24.696804    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512\": container with ID starting with 3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512 not found: ID does not exist" containerID="3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512"
Sep 09 21:04:24.696875 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.696833    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512"} err="failed to get container status \"3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512\": rpc error: code = NotFound desc = could not find container \"3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512\": container with ID starting with 3ee7b045907ac5ad34bb63ec87f8ab51f4b46c98d29d6ab73b78ee7e2ea6a512 not found: ID does not exist"
Sep 09 21:04:24.696875 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.696850    2580 scope.go:117] "RemoveContainer" containerID="f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e"
Sep 09 21:04:24.697075 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:04:24.697061    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e\": container with ID starting with f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e not found: ID does not exist" containerID="f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e"
Sep 09 21:04:24.697112 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.697078    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e"} err="failed to get container status \"f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e\": rpc error: code = NotFound desc = could not find container \"f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e\": container with ID starting with f2c9ed1f55c8387745754fb25e7d48d9f0e4e94a435fe2f1713e53dc32c9721e not found: ID does not exist"
Sep 09 21:04:24.697112 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.697091    2580 scope.go:117] "RemoveContainer" containerID="44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940"
Sep 09 21:04:24.697300 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:04:24.697287    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940\": container with ID starting with 44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940 not found: ID does not exist" containerID="44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940"
Sep 09 21:04:24.697337 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.697303    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940"} err="failed to get container status \"44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940\": rpc error: code = NotFound desc = could not find container \"44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940\": container with ID starting with 44cdb89035e2288bd0961d39ea2f4d2ac39cc9ac5036ad750a1e6537eca58940 not found: ID does not exist"
Sep 09 21:04:24.697337 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.697313    2580 scope.go:117] "RemoveContainer" containerID="d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502"
Sep 09 21:04:24.697511 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:04:24.697497    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502\": container with ID starting with d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502 not found: ID does not exist" containerID="d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502"
Sep 09 21:04:24.697557 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.697514    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502"} err="failed to get container status \"d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502\": rpc error: code = NotFound desc = could not find container \"d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502\": container with ID starting with d0a299f44f0161d740a427824e56b037c2026e1016da2f587b90494e9d552502 not found: ID does not exist"
Sep 09 21:04:24.936134 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:24.936048    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" path="/var/lib/kubelet/pods/c8e3a316-4557-4819-98ef-489e732e3cf7/volumes"
Sep 09 21:04:26.542261 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:26.542217    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:04:26.542672 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:26.542642    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:36.542296 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:36.542251    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:04:36.542808 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:36.542788    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:46.542346 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:46.542303    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:04:46.542849 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:46.542780    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:04:56.541931 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:56.541884    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:04:56.542445 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:04:56.542238    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:06.542374 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:06.542337    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:05:06.542793 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:06.542742    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:16.542973 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:16.542900    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:05:16.543320 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:16.542992    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:05:29.094533 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:29.094501    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"]
Sep 09 21:05:29.095012 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:29.094755    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" containerID="cri-o://a8a2d043792a047aad56e40d7a8b4ac9a39b66bf461793681252d41cbd850b98" gracePeriod=30
Sep 09 21:05:29.095012 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:29.094796    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" containerID="cri-o://774fa2fed42d437475d9c07f7ab84d83c3bfa08765ba320df8577c08c2e17fb1" gracePeriod=30
Sep 09 21:05:29.095012 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:29.094810    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy" containerID="cri-o://3b481d37125fa6200ef9edeff59cfdd9119599431975a062a4651970f39cc390" gracePeriod=30
Sep 09 21:05:29.987935 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:29.987897    2580 generic.go:356] "Generic (PLEG): container finished" podID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerID="3b481d37125fa6200ef9edeff59cfdd9119599431975a062a4651970f39cc390" exitCode=2
Sep 09 21:05:29.988121 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:29.987980    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerDied","Data":"3b481d37125fa6200ef9edeff59cfdd9119599431975a062a4651970f39cc390"}
Sep 09 21:05:31.535869 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:31.535825    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 09 21:05:34.009350 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:34.009318    2580 generic.go:356] "Generic (PLEG): container finished" podID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerID="a8a2d043792a047aad56e40d7a8b4ac9a39b66bf461793681252d41cbd850b98" exitCode=0
Sep 09 21:05:34.009744 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:34.009399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerDied","Data":"a8a2d043792a047aad56e40d7a8b4ac9a39b66bf461793681252d41cbd850b98"}
Sep 09 21:05:36.535977 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:36.535933    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 09 21:05:36.541959 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:36.541927    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:05:36.542285 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:36.542261    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:41.536515 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:41.536467    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 09 21:05:41.536943 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:41.536650    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:05:46.536036 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:46.535993    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 09 21:05:46.542428 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:46.542375    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:05:46.542741 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:46.542718    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:49.015808 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.015779    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"]
Sep 09 21:05:49.016175 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016137    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container"
Sep 09 21:05:49.016175 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016150    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container"
Sep 09 21:05:49.016175 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016163    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="storage-initializer"
Sep 09 21:05:49.016175 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016168    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="storage-initializer"
Sep 09 21:05:49.016175 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016174    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kube-rbac-proxy"
Sep 09 21:05:49.016175 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016179    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kube-rbac-proxy"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016188    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="storage-initializer"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016194    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="storage-initializer"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016207    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016212    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016217    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016222    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016229    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016234    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016294    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kserve-container"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016303    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="kube-rbac-proxy"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016309    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kserve-container"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016315    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8e3a316-4557-4819-98ef-489e732e3cf7" containerName="agent"
Sep 09 21:05:49.016366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.016321    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1347a1-0ad3-41a9-8dd1-c57bbe692d4e" containerName="kube-rbac-proxy"
Sep 09 21:05:49.019127 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.019112    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.022669 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.022648    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 09 21:05:49.022977 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.022959    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 09 21:05:49.043613 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.043024    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"]
Sep 09 21:05:49.109673 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.109636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd7078b9-8d38-410d-873c-4068b13e3ab3-proxy-tls\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.109673 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.109683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd7078b9-8d38-410d-873c-4068b13e3ab3-kserve-provision-location\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.109957 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.109705    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.109957 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.109778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-cabundle-cert\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.109957 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.109854    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9bq\" (UniqueName: \"kubernetes.io/projected/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-api-access-lp9bq\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.210380 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.210346    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd7078b9-8d38-410d-873c-4068b13e3ab3-proxy-tls\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.210569 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.210390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd7078b9-8d38-410d-873c-4068b13e3ab3-kserve-provision-location\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.210569 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.210412    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.210569 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.210527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-cabundle-cert\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.210745 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.210599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9bq\" (UniqueName: \"kubernetes.io/projected/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-api-access-lp9bq\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.210871 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.210851    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd7078b9-8d38-410d-873c-4068b13e3ab3-kserve-provision-location\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.211137 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.211117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.211198 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.211187    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-cabundle-cert\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.213049 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.213028    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd7078b9-8d38-410d-873c-4068b13e3ab3-proxy-tls\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.220089 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.220056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9bq\" (UniqueName: \"kubernetes.io/projected/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-api-access-lp9bq\") pod \"isvc-logger-predictor-85cd966f49-tlng5\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.329791 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.329684    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:49.614515 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:49.614485    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"]
Sep 09 21:05:49.619051 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:05:49.619024    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7078b9_8d38_410d_873c_4068b13e3ab3.slice/crio-375fb4c970d5c6dac908cfd7ab09bd13370a3cfda600e1b9e08d99f36373004d WatchSource:0}: Error finding container 375fb4c970d5c6dac908cfd7ab09bd13370a3cfda600e1b9e08d99f36373004d: Status 404 returned error can't find the container with id 375fb4c970d5c6dac908cfd7ab09bd13370a3cfda600e1b9e08d99f36373004d
Sep 09 21:05:50.088031 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:50.087932    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerStarted","Data":"ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c"}
Sep 09 21:05:50.088031 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:50.088011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerStarted","Data":"375fb4c970d5c6dac908cfd7ab09bd13370a3cfda600e1b9e08d99f36373004d"}
Sep 09 21:05:51.535954 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:51.535908    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 09 21:05:54.108901 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:54.108869    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerID="ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c" exitCode=0
Sep 09 21:05:54.109273 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:54.108941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerDied","Data":"ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c"}
Sep 09 21:05:55.117554 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:55.117519    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerStarted","Data":"c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0"}
Sep 09 21:05:55.117978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:55.117587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerStarted","Data":"45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07"}
Sep 09 21:05:55.117978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:55.117611    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerStarted","Data":"e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a"}
Sep 09 21:05:55.117978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:55.117893    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:55.118136 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:55.118029    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:55.119424 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:55.119403    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:05:55.139136 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:55.139077    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podStartSLOduration=3.270533715 podStartE2EDuration="7.139064537s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.868530822s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:55.138701788 +0000 UTC m=+844.699729106" watchObservedRunningTime="2026-09-09 21:05:55.139064537 +0000 UTC m=+844.700091854"
Sep 09 21:05:56.122500 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:56.122468    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:05:56.122911 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:56.122652    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:05:56.123830 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:56.123808    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:56.536696 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:56.536594    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Sep 09 21:05:56.542049 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:56.542010    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:5000: connect: connection refused"
Sep 09 21:05:56.542249 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:56.542232    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:05:56.542364 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:56.542345    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:56.542471 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:56.542457    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:05:57.127600 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:57.127554    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:05:57.128133 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:57.128109    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:05:59.141182 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.141150    2580 generic.go:356] "Generic (PLEG): container finished" podID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerID="774fa2fed42d437475d9c07f7ab84d83c3bfa08765ba320df8577c08c2e17fb1" exitCode=0
Sep 09 21:05:59.141499 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.141218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerDied","Data":"774fa2fed42d437475d9c07f7ab84d83c3bfa08765ba320df8577c08c2e17fb1"}
Sep 09 21:05:59.253867 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.253843    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:05:59.297348 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.297320    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b1d1e7-d848-4405-9740-e92e3331eff5-kserve-provision-location\") pod \"29b1d1e7-d848-4405-9740-e92e3331eff5\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") "
Sep 09 21:05:59.297542 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.297380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-cabundle-cert\") pod \"29b1d1e7-d848-4405-9740-e92e3331eff5\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") "
Sep 09 21:05:59.297542 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.297416    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29b1d1e7-d848-4405-9740-e92e3331eff5-proxy-tls\") pod \"29b1d1e7-d848-4405-9740-e92e3331eff5\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") "
Sep 09 21:05:59.297542 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.297483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gx9m\" (UniqueName: \"kubernetes.io/projected/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-api-access-6gx9m\") pod \"29b1d1e7-d848-4405-9740-e92e3331eff5\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") "
Sep 09 21:05:59.297711 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.297549    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-rbac-proxy-sar-config\") pod \"29b1d1e7-d848-4405-9740-e92e3331eff5\" (UID: \"29b1d1e7-d848-4405-9740-e92e3331eff5\") "
Sep 09 21:05:59.297777 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.297731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29b1d1e7-d848-4405-9740-e92e3331eff5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29b1d1e7-d848-4405-9740-e92e3331eff5" (UID: "29b1d1e7-d848-4405-9740-e92e3331eff5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:59.297837 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.297785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29b1d1e7-d848-4405-9740-e92e3331eff5" (UID: "29b1d1e7-d848-4405-9740-e92e3331eff5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:59.298084 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.298058    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "29b1d1e7-d848-4405-9740-e92e3331eff5" (UID: "29b1d1e7-d848-4405-9740-e92e3331eff5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:59.299824 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.299800    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b1d1e7-d848-4405-9740-e92e3331eff5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "29b1d1e7-d848-4405-9740-e92e3331eff5" (UID: "29b1d1e7-d848-4405-9740-e92e3331eff5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:05:59.299913 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.299856    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-api-access-6gx9m" (OuterVolumeSpecName: "kube-api-access-6gx9m") pod "29b1d1e7-d848-4405-9740-e92e3331eff5" (UID: "29b1d1e7-d848-4405-9740-e92e3331eff5"). InnerVolumeSpecName "kube-api-access-6gx9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:59.398749 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.398658    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gx9m\" (UniqueName: \"kubernetes.io/projected/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-api-access-6gx9m\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:05:59.398749 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.398689    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:05:59.398749 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.398698    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29b1d1e7-d848-4405-9740-e92e3331eff5-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:05:59.398749 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.398707    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29b1d1e7-d848-4405-9740-e92e3331eff5-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:05:59.398749 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:05:59.398717    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/29b1d1e7-d848-4405-9740-e92e3331eff5-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:06:00.149467 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:00.149444    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"
Sep 09 21:06:00.149467 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:00.149454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c" event={"ID":"29b1d1e7-d848-4405-9740-e92e3331eff5","Type":"ContainerDied","Data":"13c0213112e66c3498173a592cc0d94a6bc968221335b8a493f5b53e41da5c1f"}
Sep 09 21:06:00.149969 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:00.149533    2580 scope.go:117] "RemoveContainer" containerID="774fa2fed42d437475d9c07f7ab84d83c3bfa08765ba320df8577c08c2e17fb1"
Sep 09 21:06:00.160538 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:00.160521    2580 scope.go:117] "RemoveContainer" containerID="3b481d37125fa6200ef9edeff59cfdd9119599431975a062a4651970f39cc390"
Sep 09 21:06:00.168264 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:00.168243    2580 scope.go:117] "RemoveContainer" containerID="a8a2d043792a047aad56e40d7a8b4ac9a39b66bf461793681252d41cbd850b98"
Sep 09 21:06:00.172777 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:00.172743    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"]
Sep 09 21:06:00.176515 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:00.176495    2580 scope.go:117] "RemoveContainer" containerID="d37fef4414006bec679dd6684438eb1ea818598aa8d9df45308b4abe9249066d"
Sep 09 21:06:00.177021 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:00.176996    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-5d97c66745-bjm2c"]
Sep 09 21:06:00.937022 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:00.936990    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" path="/var/lib/kubelet/pods/29b1d1e7-d848-4405-9740-e92e3331eff5/volumes"
Sep 09 21:06:02.132015 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:02.131986    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:06:02.132533 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:02.132493    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:06:02.132839 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:02.132814    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:12.132812 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:12.132750    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:06:12.133294 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:12.133204    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:22.133250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:22.133205    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:06:22.133721 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:22.133615    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:32.133257 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:32.133216    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:06:32.133727 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:32.133648    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:42.132543 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:42.132450    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:06:42.133004 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:42.132925    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:06:50.898136 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:50.898110    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:06:50.900711 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:50.900691    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:06:52.132749 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:52.132712    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:06:52.133320 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:06:52.133296    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:02.132699 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:02.132655    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:07:02.135161 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:02.133119    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:12.134239 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:12.134187    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:07:12.135206 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:12.135182    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:07:24.112570 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:24.112533    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"]
Sep 09 21:07:24.113038 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:24.112856    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" containerID="cri-o://e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a" gracePeriod=30
Sep 09 21:07:24.113038 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:24.112898    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" containerID="cri-o://c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0" gracePeriod=30
Sep 09 21:07:24.113038 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:24.112912    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy" containerID="cri-o://45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07" gracePeriod=30
Sep 09 21:07:24.562675 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:24.562585    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerID="45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07" exitCode=2
Sep 09 21:07:24.562675 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:24.562657    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerDied","Data":"45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07"}
Sep 09 21:07:27.127965 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:27.127926    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 09 21:07:29.590361 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:29.590330    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerID="e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a" exitCode=0
Sep 09 21:07:29.590730 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:29.590399    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerDied","Data":"e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a"}
Sep 09 21:07:32.128414 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:32.128357    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 09 21:07:32.132651 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:32.132626    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:07:32.133066 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:32.133044    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:37.128740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:37.128691    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 09 21:07:37.129232 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:37.128842    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:07:42.128325 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:42.128273    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 09 21:07:42.132579 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:42.132546    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:07:42.132935 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:42.132911    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:47.127778 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:47.127724    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 09 21:07:49.414437 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.414405    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"]
Sep 09 21:07:49.414971 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.414951    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container"
Sep 09 21:07:49.415054 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.414974    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container"
Sep 09 21:07:49.415054 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.414988    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy"
Sep 09 21:07:49.415054 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.414995    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy"
Sep 09 21:07:49.415054 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.415019    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent"
Sep 09 21:07:49.415054 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.415028    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent"
Sep 09 21:07:49.415319 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.415056    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="storage-initializer"
Sep 09 21:07:49.415319 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.415064    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="storage-initializer"
Sep 09 21:07:49.415319 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.415186    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="agent"
Sep 09 21:07:49.415319 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.415200    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kserve-container"
Sep 09 21:07:49.415319 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.415213    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="29b1d1e7-d848-4405-9740-e92e3331eff5" containerName="kube-rbac-proxy"
Sep 09 21:07:49.418585 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.418564    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.420850 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.420832    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 09 21:07:49.421209 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.421193    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 09 21:07:49.428020 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.428000    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"]
Sep 09 21:07:49.430372 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.430352    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.430471 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.430388    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.430471 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.430429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.430587 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.430480    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.430587 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.430516    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2c44\" (UniqueName: \"kubernetes.io/projected/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-api-access-v2c44\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.531086 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.531045    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2c44\" (UniqueName: \"kubernetes.io/projected/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-api-access-v2c44\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.531283 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.531132    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.531283 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.531164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.531283 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.531218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.531283 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.531266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.531623 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.531601    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.531955 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.531931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.532022 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.531958    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.533897 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.533877    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.539459 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.539441    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2c44\" (UniqueName: \"kubernetes.io/projected/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-api-access-v2c44\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:49.729157 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:49.729077    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:50.003806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:50.003754    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"]
Sep 09 21:07:50.005978 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:07:50.005952    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6662e0fb_efcb_49e3_8aa1_75701f48ccc3.slice/crio-5314ada15e0e3f41180caeab680ee50ac23e9738fc89f66450637662126efff4 WatchSource:0}: Error finding container 5314ada15e0e3f41180caeab680ee50ac23e9738fc89f66450637662126efff4: Status 404 returned error can't find the container with id 5314ada15e0e3f41180caeab680ee50ac23e9738fc89f66450637662126efff4
Sep 09 21:07:50.694502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:50.694462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" event={"ID":"6662e0fb-efcb-49e3-8aa1-75701f48ccc3","Type":"ContainerStarted","Data":"dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5"}
Sep 09 21:07:50.694923 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:50.694560    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" event={"ID":"6662e0fb-efcb-49e3-8aa1-75701f48ccc3","Type":"ContainerStarted","Data":"5314ada15e0e3f41180caeab680ee50ac23e9738fc89f66450637662126efff4"}
Sep 09 21:07:52.127949 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:52.127908    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 09 21:07:52.133417 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:52.133378    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 09 21:07:52.133558 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:52.133523    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:07:52.133827 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:52.133805    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:07:52.133930 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:52.133916    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:07:54.261265 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.261239    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:07:54.371129 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.371102    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-rbac-proxy-sar-config\") pod \"fd7078b9-8d38-410d-873c-4068b13e3ab3\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") "
Sep 09 21:07:54.371298 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.371156    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd7078b9-8d38-410d-873c-4068b13e3ab3-kserve-provision-location\") pod \"fd7078b9-8d38-410d-873c-4068b13e3ab3\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") "
Sep 09 21:07:54.371298 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.371190    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd7078b9-8d38-410d-873c-4068b13e3ab3-proxy-tls\") pod \"fd7078b9-8d38-410d-873c-4068b13e3ab3\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") "
Sep 09 21:07:54.371298 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.371215    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp9bq\" (UniqueName: \"kubernetes.io/projected/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-api-access-lp9bq\") pod \"fd7078b9-8d38-410d-873c-4068b13e3ab3\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") "
Sep 09 21:07:54.371298 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.371243    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-cabundle-cert\") pod \"fd7078b9-8d38-410d-873c-4068b13e3ab3\" (UID: \"fd7078b9-8d38-410d-873c-4068b13e3ab3\") "
Sep 09 21:07:54.371512 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.371469    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd7078b9-8d38-410d-873c-4068b13e3ab3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd7078b9-8d38-410d-873c-4068b13e3ab3" (UID: "fd7078b9-8d38-410d-873c-4068b13e3ab3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:54.371569 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.371512    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fd7078b9-8d38-410d-873c-4068b13e3ab3" (UID: "fd7078b9-8d38-410d-873c-4068b13e3ab3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:54.371710 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.371685    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fd7078b9-8d38-410d-873c-4068b13e3ab3" (UID: "fd7078b9-8d38-410d-873c-4068b13e3ab3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:54.373379 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.373354    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7078b9-8d38-410d-873c-4068b13e3ab3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fd7078b9-8d38-410d-873c-4068b13e3ab3" (UID: "fd7078b9-8d38-410d-873c-4068b13e3ab3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:54.373479 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.373402    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-api-access-lp9bq" (OuterVolumeSpecName: "kube-api-access-lp9bq") pod "fd7078b9-8d38-410d-873c-4068b13e3ab3" (UID: "fd7078b9-8d38-410d-873c-4068b13e3ab3"). InnerVolumeSpecName "kube-api-access-lp9bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:54.472192 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.472159    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd7078b9-8d38-410d-873c-4068b13e3ab3-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:07:54.472192 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.472185    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fd7078b9-8d38-410d-873c-4068b13e3ab3-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:07:54.472192 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.472195    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9bq\" (UniqueName: \"kubernetes.io/projected/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-api-access-lp9bq\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:07:54.472426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.472204    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:07:54.472426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.472213    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fd7078b9-8d38-410d-873c-4068b13e3ab3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:07:54.723615 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.723522    2580 generic.go:356] "Generic (PLEG): container finished" podID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerID="c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0" exitCode=0
Sep 09 21:07:54.723615 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.723598    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"
Sep 09 21:07:54.723615 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.723601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerDied","Data":"c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0"}
Sep 09 21:07:54.723872 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.723698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5" event={"ID":"fd7078b9-8d38-410d-873c-4068b13e3ab3","Type":"ContainerDied","Data":"375fb4c970d5c6dac908cfd7ab09bd13370a3cfda600e1b9e08d99f36373004d"}
Sep 09 21:07:54.723872 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.723735    2580 scope.go:117] "RemoveContainer" containerID="c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0"
Sep 09 21:07:54.725383 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.725359    2580 generic.go:356] "Generic (PLEG): container finished" podID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerID="dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5" exitCode=0
Sep 09 21:07:54.725487 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.725388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" event={"ID":"6662e0fb-efcb-49e3-8aa1-75701f48ccc3","Type":"ContainerDied","Data":"dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5"}
Sep 09 21:07:54.733674 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.733658    2580 scope.go:117] "RemoveContainer" containerID="45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07"
Sep 09 21:07:54.742066 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.742047    2580 scope.go:117] "RemoveContainer" containerID="e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a"
Sep 09 21:07:54.751931 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.751900    2580 scope.go:117] "RemoveContainer" containerID="ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c"
Sep 09 21:07:54.762850 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.762828    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"]
Sep 09 21:07:54.765120 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.765103    2580 scope.go:117] "RemoveContainer" containerID="c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0"
Sep 09 21:07:54.765463 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:07:54.765443    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0\": container with ID starting with c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0 not found: ID does not exist" containerID="c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0"
Sep 09 21:07:54.765548 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.765475    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0"} err="failed to get container status \"c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0\": rpc error: code = NotFound desc = could not find container \"c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0\": container with ID starting with c5a96962f107a81c9a079ba9449a87bd42fa98974b9d91567f1e73b089c2e2a0 not found: ID does not exist"
Sep 09 21:07:54.765548 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.765497    2580 scope.go:117] "RemoveContainer" containerID="45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07"
Sep 09 21:07:54.765842 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:07:54.765824    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07\": container with ID starting with 45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07 not found: ID does not exist" containerID="45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07"
Sep 09 21:07:54.765900 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.765848    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07"} err="failed to get container status \"45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07\": rpc error: code = NotFound desc = could not find container \"45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07\": container with ID starting with 45d04e9d454a5afe82926b2490630ffa83d7898ed83643dd02bcca0f873f8d07 not found: ID does not exist"
Sep 09 21:07:54.765900 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.765863    2580 scope.go:117] "RemoveContainer" containerID="e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a"
Sep 09 21:07:54.766134 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:07:54.766109    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a\": container with ID starting with e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a not found: ID does not exist" containerID="e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a"
Sep 09 21:07:54.766202 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.766143    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a"} err="failed to get container status \"e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a\": rpc error: code = NotFound desc = could not find container \"e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a\": container with ID starting with e0f545f0567a0c14e54a14abb3fb95e4894b5b96e67656ac281246d705349e6a not found: ID does not exist"
Sep 09 21:07:54.766202 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.766164    2580 scope.go:117] "RemoveContainer" containerID="ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c"
Sep 09 21:07:54.766486 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:07:54.766461    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c\": container with ID starting with ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c not found: ID does not exist" containerID="ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c"
Sep 09 21:07:54.766567 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.766494    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c"} err="failed to get container status \"ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c\": rpc error: code = NotFound desc = could not find container \"ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c\": container with ID starting with ff2e01e33518480c336920924f7caae4e72315b9a21333b52f14afde7a85408c not found: ID does not exist"
Sep 09 21:07:54.769097 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.769074    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-85cd966f49-tlng5"]
Sep 09 21:07:54.936457 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:54.936429    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" path="/var/lib/kubelet/pods/fd7078b9-8d38-410d-873c-4068b13e3ab3/volumes"
Sep 09 21:07:55.733392 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:55.733362    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" event={"ID":"6662e0fb-efcb-49e3-8aa1-75701f48ccc3","Type":"ContainerStarted","Data":"1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f"}
Sep 09 21:07:55.733798 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:55.733417    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" event={"ID":"6662e0fb-efcb-49e3-8aa1-75701f48ccc3","Type":"ContainerStarted","Data":"04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac"}
Sep 09 21:07:55.733798 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:55.733610    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:07:55.755022 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:55.754975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podStartSLOduration=2.773530807 podStartE2EDuration="6.754959985s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.981429178s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:07:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:07:55.753013542 +0000 UTC m=+965.314040870" watchObservedRunningTime="2026-09-09 21:07:55.754959985 +0000 UTC m=+965.315987301"
Sep 09 21:07:56.739387 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:07:56.739358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:08:02.748194 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:08:02.748165    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:08:32.749988 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:08:32.749908    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 21:08:42.749398 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:08:42.749358    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 21:08:52.749016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:08:52.748980    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 21:09:02.749652 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:02.749613    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 21:09:03.930455 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:03.930409    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 21:09:13.956080 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:13.956040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:09:19.529260 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:19.529220    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"]
Sep 09 21:09:19.529676 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:19.529513    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container" containerID="cri-o://04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac" gracePeriod=30
Sep 09 21:09:19.529676 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:19.529519    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kube-rbac-proxy" containerID="cri-o://1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f" gracePeriod=30
Sep 09 21:09:20.136830 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:20.136797    2580 generic.go:356] "Generic (PLEG): container finished" podID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerID="1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f" exitCode=2
Sep 09 21:09:20.136830 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:20.136834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" event={"ID":"6662e0fb-efcb-49e3-8aa1-75701f48ccc3","Type":"ContainerDied","Data":"1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f"}
Sep 09 21:09:22.743985 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:22.743943    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 09 21:09:23.930536 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:23.930501    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.43:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 09 21:09:24.478655 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.478625    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:09:24.621118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.621090    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2c44\" (UniqueName: \"kubernetes.io/projected/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-api-access-v2c44\") pod \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") "
Sep 09 21:09:24.621280 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.621135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kserve-provision-location\") pod \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") "
Sep 09 21:09:24.621280 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.621195    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-rbac-proxy-sar-config\") pod \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") "
Sep 09 21:09:24.621280 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.621233    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-proxy-tls\") pod \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") "
Sep 09 21:09:24.621280 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.621268    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-cabundle-cert\") pod \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\" (UID: \"6662e0fb-efcb-49e3-8aa1-75701f48ccc3\") "
Sep 09 21:09:24.621552 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.621524    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6662e0fb-efcb-49e3-8aa1-75701f48ccc3" (UID: "6662e0fb-efcb-49e3-8aa1-75701f48ccc3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:24.621660 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.621555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6662e0fb-efcb-49e3-8aa1-75701f48ccc3" (UID: "6662e0fb-efcb-49e3-8aa1-75701f48ccc3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:24.621817 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.621715    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6662e0fb-efcb-49e3-8aa1-75701f48ccc3" (UID: "6662e0fb-efcb-49e3-8aa1-75701f48ccc3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:24.623429 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.623395    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6662e0fb-efcb-49e3-8aa1-75701f48ccc3" (UID: "6662e0fb-efcb-49e3-8aa1-75701f48ccc3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:24.623537 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.623437    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-api-access-v2c44" (OuterVolumeSpecName: "kube-api-access-v2c44") pod "6662e0fb-efcb-49e3-8aa1-75701f48ccc3" (UID: "6662e0fb-efcb-49e3-8aa1-75701f48ccc3"). InnerVolumeSpecName "kube-api-access-v2c44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:24.704311 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704274    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"]
Sep 09 21:09:24.704650 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704638    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container"
Sep 09 21:09:24.704705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704652    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container"
Sep 09 21:09:24.704705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704664    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="storage-initializer"
Sep 09 21:09:24.704705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704670    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="storage-initializer"
Sep 09 21:09:24.704705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704684    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container"
Sep 09 21:09:24.704705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704689    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container"
Sep 09 21:09:24.704705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704698    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="storage-initializer"
Sep 09 21:09:24.704705 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704702    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="storage-initializer"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704709    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704714    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704720    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704725    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704731    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kube-rbac-proxy"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704735    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kube-rbac-proxy"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704818    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kube-rbac-proxy"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704833    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="agent"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704843    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerName="kserve-container"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704852    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kserve-container"
Sep 09 21:09:24.704948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.704857    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd7078b9-8d38-410d-873c-4068b13e3ab3" containerName="kube-rbac-proxy"
Sep 09 21:09:24.708131 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.708115    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.710150 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.710130    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 09 21:09:24.712371 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.712355    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 09 21:09:24.719207 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.719187    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"]
Sep 09 21:09:24.722526 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.722508    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:09:24.722526 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.722526    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:09:24.722644 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.722536    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2c44\" (UniqueName: \"kubernetes.io/projected/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-api-access-v2c44\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:09:24.722644 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.722546    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:09:24.722644 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.722557    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6662e0fb-efcb-49e3-8aa1-75701f48ccc3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:09:24.823397 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.823359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.823576 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.823405    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4d919b2-8361-4889-8748-9b9170d211fb-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.823576 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.823437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4d919b2-8361-4889-8748-9b9170d211fb-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.823576 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.823467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.823576 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.823488    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rzdb\" (UniqueName: \"kubernetes.io/projected/d4d919b2-8361-4889-8748-9b9170d211fb-kube-api-access-5rzdb\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.924876 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.924785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4d919b2-8361-4889-8748-9b9170d211fb-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.924876 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.924843    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4d919b2-8361-4889-8748-9b9170d211fb-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.925118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.924889    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.925118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.924921    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rzdb\" (UniqueName: \"kubernetes.io/projected/d4d919b2-8361-4889-8748-9b9170d211fb-kube-api-access-5rzdb\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.925118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.924983    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.925283 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.925224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4d919b2-8361-4889-8748-9b9170d211fb-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.925629 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.925604    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.925692 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.925632    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.927479 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.927456    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4d919b2-8361-4889-8748-9b9170d211fb-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:24.935971 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:24.935945    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rzdb\" (UniqueName: \"kubernetes.io/projected/d4d919b2-8361-4889-8748-9b9170d211fb-kube-api-access-5rzdb\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:25.019253 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.019210    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:25.169259 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.169227    2580 generic.go:356] "Generic (PLEG): container finished" podID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" containerID="04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac" exitCode=0
Sep 09 21:09:25.169423 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.169288    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"
Sep 09 21:09:25.169423 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.169320    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" event={"ID":"6662e0fb-efcb-49e3-8aa1-75701f48ccc3","Type":"ContainerDied","Data":"04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac"}
Sep 09 21:09:25.169423 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.169381    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s" event={"ID":"6662e0fb-efcb-49e3-8aa1-75701f48ccc3","Type":"ContainerDied","Data":"5314ada15e0e3f41180caeab680ee50ac23e9738fc89f66450637662126efff4"}
Sep 09 21:09:25.169423 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.169409    2580 scope.go:117] "RemoveContainer" containerID="1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f"
Sep 09 21:09:25.177724 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.177696    2580 scope.go:117] "RemoveContainer" containerID="04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac"
Sep 09 21:09:25.185592 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.185572    2580 scope.go:117] "RemoveContainer" containerID="dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5"
Sep 09 21:09:25.191435 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.191411    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"]
Sep 09 21:09:25.193683 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.193669    2580 scope.go:117] "RemoveContainer" containerID="1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f"
Sep 09 21:09:25.194029 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:09:25.193998    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f\": container with ID starting with 1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f not found: ID does not exist" containerID="1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f"
Sep 09 21:09:25.194128 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.194030    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f"} err="failed to get container status \"1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f\": rpc error: code = NotFound desc = could not find container \"1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f\": container with ID starting with 1233eee2d88c7282c97273d7b8d87a9ebc4d11b939aa6983e026fda7fcc2232f not found: ID does not exist"
Sep 09 21:09:25.194128 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.194053    2580 scope.go:117] "RemoveContainer" containerID="04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac"
Sep 09 21:09:25.194381 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:09:25.194360    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac\": container with ID starting with 04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac not found: ID does not exist" containerID="04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac"
Sep 09 21:09:25.194479 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.194389    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac"} err="failed to get container status \"04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac\": rpc error: code = NotFound desc = could not find container \"04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac\": container with ID starting with 04267def6817c77f4e3aa1f75556a8edab38a17380180afacae176d18d9163ac not found: ID does not exist"
Sep 09 21:09:25.194479 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.194409    2580 scope.go:117] "RemoveContainer" containerID="dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5"
Sep 09 21:09:25.194748 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:09:25.194728    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5\": container with ID starting with dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5 not found: ID does not exist" containerID="dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5"
Sep 09 21:09:25.194885 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.194795    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5"} err="failed to get container status \"dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5\": rpc error: code = NotFound desc = could not find container \"dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5\": container with ID starting with dc6a0f550110fa04f3ce8119990343a772c6c546ea124dcc089d4c21ca0889b5 not found: ID does not exist"
Sep 09 21:09:25.196085 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.196066    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-z9f6s"]
Sep 09 21:09:25.297022 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:25.296992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"]
Sep 09 21:09:25.300389 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:09:25.300358    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d919b2_8361_4889_8748_9b9170d211fb.slice/crio-c5e20dad05aec2002a4d60b05201f2acc57610a4a97977da13bf410516785689 WatchSource:0}: Error finding container c5e20dad05aec2002a4d60b05201f2acc57610a4a97977da13bf410516785689: Status 404 returned error can't find the container with id c5e20dad05aec2002a4d60b05201f2acc57610a4a97977da13bf410516785689
Sep 09 21:09:26.175553 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:26.175518    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" event={"ID":"d4d919b2-8361-4889-8748-9b9170d211fb","Type":"ContainerStarted","Data":"7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76"}
Sep 09 21:09:26.175983 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:26.175585    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" event={"ID":"d4d919b2-8361-4889-8748-9b9170d211fb","Type":"ContainerStarted","Data":"c5e20dad05aec2002a4d60b05201f2acc57610a4a97977da13bf410516785689"}
Sep 09 21:09:26.935253 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:26.935218    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6662e0fb-efcb-49e3-8aa1-75701f48ccc3" path="/var/lib/kubelet/pods/6662e0fb-efcb-49e3-8aa1-75701f48ccc3/volumes"
Sep 09 21:09:30.196809 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:30.196748    2580 generic.go:356] "Generic (PLEG): container finished" podID="d4d919b2-8361-4889-8748-9b9170d211fb" containerID="7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76" exitCode=0
Sep 09 21:09:30.197286 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:30.196834    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" event={"ID":"d4d919b2-8361-4889-8748-9b9170d211fb","Type":"ContainerDied","Data":"7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76"}
Sep 09 21:09:31.203430 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:31.203396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" event={"ID":"d4d919b2-8361-4889-8748-9b9170d211fb","Type":"ContainerStarted","Data":"8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097"}
Sep 09 21:09:31.203818 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:31.203458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" event={"ID":"d4d919b2-8361-4889-8748-9b9170d211fb","Type":"ContainerStarted","Data":"10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812"}
Sep 09 21:09:31.203818 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:31.203782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:31.229364 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:31.229318    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" podStartSLOduration=3.386767236 podStartE2EDuration="7.22930198s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.842534744s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:31.227653785 +0000 UTC m=+1060.788681101" watchObservedRunningTime="2026-09-09 21:09:31.22930198 +0000 UTC m=+1060.790329297"
Sep 09 21:09:32.208016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:32.207981    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:09:38.216970 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:09:38.216941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:10:08.218304 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:08.218263    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 21:10:18.218355 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:18.218319    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 21:10:28.217488 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:28.217447    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 21:10:38.218344 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:38.218310    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 21:10:48.257124 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:48.257090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:10:54.728528 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:54.728496    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"]
Sep 09 21:10:54.729002 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:54.728868    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kserve-container" containerID="cri-o://10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812" gracePeriod=30
Sep 09 21:10:54.729002 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:54.728885    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kube-rbac-proxy" containerID="cri-o://8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097" gracePeriod=30
Sep 09 21:10:55.599945 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:55.599913    2580 generic.go:356] "Generic (PLEG): container finished" podID="d4d919b2-8361-4889-8748-9b9170d211fb" containerID="8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097" exitCode=2
Sep 09 21:10:55.600135 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:55.599987    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" event={"ID":"d4d919b2-8361-4889-8748-9b9170d211fb","Type":"ContainerDied","Data":"8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097"}
Sep 09 21:10:58.213038 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:58.212997    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 09 21:10:58.217868 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:58.217845    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.44:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 09 21:10:59.474433 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.474410    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:10:59.517723 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.517647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rzdb\" (UniqueName: \"kubernetes.io/projected/d4d919b2-8361-4889-8748-9b9170d211fb-kube-api-access-5rzdb\") pod \"d4d919b2-8361-4889-8748-9b9170d211fb\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") "
Sep 09 21:10:59.517874 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.517733    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4d919b2-8361-4889-8748-9b9170d211fb-proxy-tls\") pod \"d4d919b2-8361-4889-8748-9b9170d211fb\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") "
Sep 09 21:10:59.517874 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.517782    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-kube-rbac-proxy-sar-config\") pod \"d4d919b2-8361-4889-8748-9b9170d211fb\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") "
Sep 09 21:10:59.517874 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.517829    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-cabundle-cert\") pod \"d4d919b2-8361-4889-8748-9b9170d211fb\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") "
Sep 09 21:10:59.518043 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.517897    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4d919b2-8361-4889-8748-9b9170d211fb-kserve-provision-location\") pod \"d4d919b2-8361-4889-8748-9b9170d211fb\" (UID: \"d4d919b2-8361-4889-8748-9b9170d211fb\") "
Sep 09 21:10:59.518242 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.518209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d919b2-8361-4889-8748-9b9170d211fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4d919b2-8361-4889-8748-9b9170d211fb" (UID: "d4d919b2-8361-4889-8748-9b9170d211fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:59.518242 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.518228    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4d919b2-8361-4889-8748-9b9170d211fb" (UID: "d4d919b2-8361-4889-8748-9b9170d211fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:59.518604 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.518242    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4d919b2-8361-4889-8748-9b9170d211fb" (UID: "d4d919b2-8361-4889-8748-9b9170d211fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:10:59.520016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.519989    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d919b2-8361-4889-8748-9b9170d211fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4d919b2-8361-4889-8748-9b9170d211fb" (UID: "d4d919b2-8361-4889-8748-9b9170d211fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:59.520016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.519995    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d919b2-8361-4889-8748-9b9170d211fb-kube-api-access-5rzdb" (OuterVolumeSpecName: "kube-api-access-5rzdb") pod "d4d919b2-8361-4889-8748-9b9170d211fb" (UID: "d4d919b2-8361-4889-8748-9b9170d211fb"). InnerVolumeSpecName "kube-api-access-5rzdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:59.619482 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.619445    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4d919b2-8361-4889-8748-9b9170d211fb-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:10:59.619482 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.619470    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:10:59.619482 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.619480    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4d919b2-8361-4889-8748-9b9170d211fb-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:10:59.619482 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.619490    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4d919b2-8361-4889-8748-9b9170d211fb-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:10:59.619854 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.619498    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rzdb\" (UniqueName: \"kubernetes.io/projected/d4d919b2-8361-4889-8748-9b9170d211fb-kube-api-access-5rzdb\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:10:59.621708 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.621681    2580 generic.go:356] "Generic (PLEG): container finished" podID="d4d919b2-8361-4889-8748-9b9170d211fb" containerID="10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812" exitCode=0
Sep 09 21:10:59.621861 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.621719    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" event={"ID":"d4d919b2-8361-4889-8748-9b9170d211fb","Type":"ContainerDied","Data":"10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812"}
Sep 09 21:10:59.621861 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.621750    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"
Sep 09 21:10:59.621861 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.621846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf" event={"ID":"d4d919b2-8361-4889-8748-9b9170d211fb","Type":"ContainerDied","Data":"c5e20dad05aec2002a4d60b05201f2acc57610a4a97977da13bf410516785689"}
Sep 09 21:10:59.622021 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.621888    2580 scope.go:117] "RemoveContainer" containerID="8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097"
Sep 09 21:10:59.630853 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.630837    2580 scope.go:117] "RemoveContainer" containerID="10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812"
Sep 09 21:10:59.641001 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.640982    2580 scope.go:117] "RemoveContainer" containerID="7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76"
Sep 09 21:10:59.643809 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.643784    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"]
Sep 09 21:10:59.649325 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.649292    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wbtkf"]
Sep 09 21:10:59.649416 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.649328    2580 scope.go:117] "RemoveContainer" containerID="8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097"
Sep 09 21:10:59.649648 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:10:59.649621    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097\": container with ID starting with 8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097 not found: ID does not exist" containerID="8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097"
Sep 09 21:10:59.649709 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.649658    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097"} err="failed to get container status \"8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097\": rpc error: code = NotFound desc = could not find container \"8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097\": container with ID starting with 8830f0980a9b525e8d9ed88e740d6d452350f1f827bc8084eae10417f927f097 not found: ID does not exist"
Sep 09 21:10:59.649709 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.649675    2580 scope.go:117] "RemoveContainer" containerID="10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812"
Sep 09 21:10:59.650041 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:10:59.650022    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812\": container with ID starting with 10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812 not found: ID does not exist" containerID="10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812"
Sep 09 21:10:59.650111 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.650046    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812"} err="failed to get container status \"10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812\": rpc error: code = NotFound desc = could not find container \"10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812\": container with ID starting with 10395f32ed34318d8825b82f5561cd37c429afcb250283002c10736c3bfb0812 not found: ID does not exist"
Sep 09 21:10:59.650111 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.650059    2580 scope.go:117] "RemoveContainer" containerID="7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76"
Sep 09 21:10:59.650377 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:10:59.650357    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76\": container with ID starting with 7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76 not found: ID does not exist" containerID="7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76"
Sep 09 21:10:59.650445 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.650384    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76"} err="failed to get container status \"7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76\": rpc error: code = NotFound desc = could not find container \"7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76\": container with ID starting with 7d65e20bdc2792156d24decda70b4138b81ef98102e2f8fbd86790e25e028e76 not found: ID does not exist"
Sep 09 21:10:59.915694 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.915659    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"]
Sep 09 21:10:59.916128 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.916110    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kserve-container"
Sep 09 21:10:59.916226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.916130    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kserve-container"
Sep 09 21:10:59.916226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.916143    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kube-rbac-proxy"
Sep 09 21:10:59.916226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.916151    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kube-rbac-proxy"
Sep 09 21:10:59.916226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.916163    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="storage-initializer"
Sep 09 21:10:59.916226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.916173    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="storage-initializer"
Sep 09 21:10:59.916475 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.916253    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kube-rbac-proxy"
Sep 09 21:10:59.916475 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.916267    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" containerName="kserve-container"
Sep 09 21:10:59.921010 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.920991    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:10:59.925064 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.925041    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 09 21:10:59.925064 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.925055    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:10:59.925247 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.925054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 09 21:10:59.925247 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.925054    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:10:59.925247 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.925061    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:10:59.925247 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.925158    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:10:59.925247 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.925225    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:10:59.932801 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:10:59.932780    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"]
Sep 09 21:11:00.022006 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.021965    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.022006 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.022004    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.022216 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.022075    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gxsf\" (UniqueName: \"kubernetes.io/projected/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-api-access-2gxsf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.022216 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.022094    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.022216 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.022159    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb38c21e-fd68-49e4-94be-1a6d1be0246e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.123353 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.123313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.123353 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.123353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.123600 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.123398    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gxsf\" (UniqueName: \"kubernetes.io/projected/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-api-access-2gxsf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.123600 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.123420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.123600 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.123452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb38c21e-fd68-49e4-94be-1a6d1be0246e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.123600 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:11:00.123559    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 09 21:11:00.123796 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:11:00.123626    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb38c21e-fd68-49e4-94be-1a6d1be0246e-proxy-tls podName:fb38c21e-fd68-49e4-94be-1a6d1be0246e nodeName:}" failed. No retries permitted until 2026-09-09 21:11:00.623607487 +0000 UTC m=+1150.184634801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fb38c21e-fd68-49e4-94be-1a6d1be0246e-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" (UID: "fb38c21e-fd68-49e4-94be-1a6d1be0246e") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 09 21:11:00.123908 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.123886    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.124159 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.124142    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.124207 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.124145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.132151 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.132130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gxsf\" (UniqueName: \"kubernetes.io/projected/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-api-access-2gxsf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.627859 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.627827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb38c21e-fd68-49e4-94be-1a6d1be0246e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.630427 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.630409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb38c21e-fd68-49e4-94be-1a6d1be0246e-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.831630 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.831589    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:00.936231 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:00.936199    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4d919b2-8361-4889-8748-9b9170d211fb" path="/var/lib/kubelet/pods/d4d919b2-8361-4889-8748-9b9170d211fb/volumes"
Sep 09 21:11:01.104949 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:01.104925    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"]
Sep 09 21:11:01.107487 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:11:01.107458    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb38c21e_fd68_49e4_94be_1a6d1be0246e.slice/crio-ca09a760e996f116ee69b2a8649d79646c77aded8f636daa8c2092e840b4458e WatchSource:0}: Error finding container ca09a760e996f116ee69b2a8649d79646c77aded8f636daa8c2092e840b4458e: Status 404 returned error can't find the container with id ca09a760e996f116ee69b2a8649d79646c77aded8f636daa8c2092e840b4458e
Sep 09 21:11:01.635380 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:01.635343    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" event={"ID":"fb38c21e-fd68-49e4-94be-1a6d1be0246e","Type":"ContainerStarted","Data":"3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd"}
Sep 09 21:11:01.635861 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:01.635403    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" event={"ID":"fb38c21e-fd68-49e4-94be-1a6d1be0246e","Type":"ContainerStarted","Data":"ca09a760e996f116ee69b2a8649d79646c77aded8f636daa8c2092e840b4458e"}
Sep 09 21:11:05.655336 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:05.655303    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerID="3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd" exitCode=0
Sep 09 21:11:05.655791 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:05.655389    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" event={"ID":"fb38c21e-fd68-49e4-94be-1a6d1be0246e","Type":"ContainerDied","Data":"3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd"}
Sep 09 21:11:06.663618 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:06.663582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" event={"ID":"fb38c21e-fd68-49e4-94be-1a6d1be0246e","Type":"ContainerStarted","Data":"216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e"}
Sep 09 21:11:06.664165 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:06.663658    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" event={"ID":"fb38c21e-fd68-49e4-94be-1a6d1be0246e","Type":"ContainerStarted","Data":"1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99"}
Sep 09 21:11:06.664165 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:06.663932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:06.683443 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:06.683383    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" podStartSLOduration=3.673566633 podStartE2EDuration="7.683365513s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.00979888s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:06.682633505 +0000 UTC m=+1156.243660821" watchObservedRunningTime="2026-09-09 21:11:06.683365513 +0000 UTC m=+1156.244392831"
Sep 09 21:11:07.670635 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:07.670593    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:13.681844 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:13.681753    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:11:43.683391 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:43.683351    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 21:11:50.928143 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:50.928112    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:11:50.938286 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:50.938262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:11:53.682797 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:11:53.682737    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 21:12:03.683050 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:03.683011    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 21:12:13.682406 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:13.682358    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 21:12:21.956475 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:21.956434    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:12:30.011197 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:30.011164    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"]
Sep 09 21:12:30.011626 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:30.011411    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kserve-container" containerID="cri-o://1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99" gracePeriod=30
Sep 09 21:12:30.011626 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:30.011449    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kube-rbac-proxy" containerID="cri-o://216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e" gracePeriod=30
Sep 09 21:12:31.069464 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:31.069434    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerID="216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e" exitCode=2
Sep 09 21:12:31.069865 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:31.069482    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" event={"ID":"fb38c21e-fd68-49e4-94be-1a6d1be0246e","Type":"ContainerDied","Data":"216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e"}
Sep 09 21:12:31.931029 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:31.930992    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.45:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 09 21:12:33.676941 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:33.676892    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 09 21:12:35.951662 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:35.951637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:12:36.094753 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.094662    2580 generic.go:356] "Generic (PLEG): container finished" podID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerID="1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99" exitCode=0
Sep 09 21:12:36.094753 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.094712    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"
Sep 09 21:12:36.094978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.094785    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" event={"ID":"fb38c21e-fd68-49e4-94be-1a6d1be0246e","Type":"ContainerDied","Data":"1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99"}
Sep 09 21:12:36.094978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.094848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79" event={"ID":"fb38c21e-fd68-49e4-94be-1a6d1be0246e","Type":"ContainerDied","Data":"ca09a760e996f116ee69b2a8649d79646c77aded8f636daa8c2092e840b4458e"}
Sep 09 21:12:36.094978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.094877    2580 scope.go:117] "RemoveContainer" containerID="216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e"
Sep 09 21:12:36.103181 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.103165    2580 scope.go:117] "RemoveContainer" containerID="1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99"
Sep 09 21:12:36.109290 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109271    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-cabundle-cert\") pod \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") "
Sep 09 21:12:36.109389 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kserve-provision-location\") pod \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") "
Sep 09 21:12:36.109457 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-rbac-proxy-sar-config\") pod \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") "
Sep 09 21:12:36.109510 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109468    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb38c21e-fd68-49e4-94be-1a6d1be0246e-proxy-tls\") pod \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") "
Sep 09 21:12:36.109564 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gxsf\" (UniqueName: \"kubernetes.io/projected/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-api-access-2gxsf\") pod \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\" (UID: \"fb38c21e-fd68-49e4-94be-1a6d1be0246e\") "
Sep 09 21:12:36.109637 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109605    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fb38c21e-fd68-49e4-94be-1a6d1be0246e" (UID: "fb38c21e-fd68-49e4-94be-1a6d1be0246e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:36.109736 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109713    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fb38c21e-fd68-49e4-94be-1a6d1be0246e" (UID: "fb38c21e-fd68-49e4-94be-1a6d1be0246e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:12:36.109830 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109791    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fb38c21e-fd68-49e4-94be-1a6d1be0246e" (UID: "fb38c21e-fd68-49e4-94be-1a6d1be0246e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:12:36.109830 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109826    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:12:36.109948 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.109839    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:12:36.111285 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.111263    2580 scope.go:117] "RemoveContainer" containerID="3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd"
Sep 09 21:12:36.111661 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.111637    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb38c21e-fd68-49e4-94be-1a6d1be0246e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fb38c21e-fd68-49e4-94be-1a6d1be0246e" (UID: "fb38c21e-fd68-49e4-94be-1a6d1be0246e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:36.112034 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.112009    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-api-access-2gxsf" (OuterVolumeSpecName: "kube-api-access-2gxsf") pod "fb38c21e-fd68-49e4-94be-1a6d1be0246e" (UID: "fb38c21e-fd68-49e4-94be-1a6d1be0246e"). InnerVolumeSpecName "kube-api-access-2gxsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:36.126885 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.126862    2580 scope.go:117] "RemoveContainer" containerID="216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e"
Sep 09 21:12:36.127180 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:12:36.127161    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e\": container with ID starting with 216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e not found: ID does not exist" containerID="216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e"
Sep 09 21:12:36.127233 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.127190    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e"} err="failed to get container status \"216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e\": rpc error: code = NotFound desc = could not find container \"216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e\": container with ID starting with 216c583133f846ea074f29bf6f46d3a4cf02849f91b86172727ece737bcbb77e not found: ID does not exist"
Sep 09 21:12:36.127233 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.127208    2580 scope.go:117] "RemoveContainer" containerID="1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99"
Sep 09 21:12:36.127491 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:12:36.127471    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99\": container with ID starting with 1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99 not found: ID does not exist" containerID="1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99"
Sep 09 21:12:36.127534 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.127499    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99"} err="failed to get container status \"1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99\": rpc error: code = NotFound desc = could not find container \"1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99\": container with ID starting with 1f6cd14fccdb83b9c5516c620a8a69e6d4ae2b46398e9892a2a2b33dff936b99 not found: ID does not exist"
Sep 09 21:12:36.127534 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.127518    2580 scope.go:117] "RemoveContainer" containerID="3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd"
Sep 09 21:12:36.127791 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:12:36.127741    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd\": container with ID starting with 3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd not found: ID does not exist" containerID="3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd"
Sep 09 21:12:36.127856 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.127794    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd"} err="failed to get container status \"3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd\": rpc error: code = NotFound desc = could not find container \"3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd\": container with ID starting with 3115221cae179c84fbe7e618883f62226e2784bc9daeff4088d55ed1477fc4cd not found: ID does not exist"
Sep 09 21:12:36.210486 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.210453    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:12:36.210486 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.210481    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fb38c21e-fd68-49e4-94be-1a6d1be0246e-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:12:36.210486 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.210494    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gxsf\" (UniqueName: \"kubernetes.io/projected/fb38c21e-fd68-49e4-94be-1a6d1be0246e-kube-api-access-2gxsf\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:12:36.417254 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.417227    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"]
Sep 09 21:12:36.423061 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.423035    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-w8x79"]
Sep 09 21:12:36.935577 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:12:36.935543    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" path="/var/lib/kubelet/pods/fb38c21e-fd68-49e4-94be-1a6d1be0246e/volumes"
Sep 09 21:13:09.675716 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.675678    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"]
Sep 09 21:13:09.676195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.676098    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kserve-container"
Sep 09 21:13:09.676195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.676111    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kserve-container"
Sep 09 21:13:09.676195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.676124    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="storage-initializer"
Sep 09 21:13:09.676195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.676130    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="storage-initializer"
Sep 09 21:13:09.676195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.676136    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kube-rbac-proxy"
Sep 09 21:13:09.676195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.676142    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kube-rbac-proxy"
Sep 09 21:13:09.676195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.676191    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kube-rbac-proxy"
Sep 09 21:13:09.676195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.676199    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb38c21e-fd68-49e4-94be-1a6d1be0246e" containerName="kserve-container"
Sep 09 21:13:09.680589 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.680571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.682890 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.682866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:13:09.683003 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.682866    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 09 21:13:09.683073 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.682996    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:13:09.683073 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.682925    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:13:09.683185 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.683140    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 09 21:13:09.683185 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.683147    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:13:09.683538 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.683519    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:13:09.689651 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.689630    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"]
Sep 09 21:13:09.793061 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.793026    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.793259 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.793084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-658p7\" (UniqueName: \"kubernetes.io/projected/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-api-access-658p7\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.793259 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.793166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-proxy-tls\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.793259 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.793192    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-cabundle-cert\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.793259 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.793214    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.894416 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.894380    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.894612 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.894428    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-658p7\" (UniqueName: \"kubernetes.io/projected/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-api-access-658p7\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.894612 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.894479    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-proxy-tls\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.894612 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.894503    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-cabundle-cert\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.894612 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.894526    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.894959 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.894939    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.895153 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.895130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.895198 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.895163    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-cabundle-cert\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.897186 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.897164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-proxy-tls\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.902354 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.902336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-658p7\" (UniqueName: \"kubernetes.io/projected/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-api-access-658p7\") pod \"isvc-sklearn-predictor-5bd8597c94-nk7fg\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:09.992543 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:09.992453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:10.268333 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:10.268311    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"]
Sep 09 21:13:10.270380 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:13:10.270351    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc8c6d21_5fc6_4e1d_b0c2_7ad7bc3541d3.slice/crio-288d6f452cef7c16d8a72c51c1110669e7a8af4d088f78c37945a5aadf54969e WatchSource:0}: Error finding container 288d6f452cef7c16d8a72c51c1110669e7a8af4d088f78c37945a5aadf54969e: Status 404 returned error can't find the container with id 288d6f452cef7c16d8a72c51c1110669e7a8af4d088f78c37945a5aadf54969e
Sep 09 21:13:11.267647 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:11.267604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" event={"ID":"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3","Type":"ContainerStarted","Data":"d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae"}
Sep 09 21:13:11.268087 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:11.267678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" event={"ID":"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3","Type":"ContainerStarted","Data":"288d6f452cef7c16d8a72c51c1110669e7a8af4d088f78c37945a5aadf54969e"}
Sep 09 21:13:15.288006 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:15.287963    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerID="d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae" exitCode=0
Sep 09 21:13:15.288573 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:15.288013    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" event={"ID":"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3","Type":"ContainerDied","Data":"d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae"}
Sep 09 21:13:16.295378 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:16.295342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" event={"ID":"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3","Type":"ContainerStarted","Data":"ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010"}
Sep 09 21:13:16.295794 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:16.295404    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" event={"ID":"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3","Type":"ContainerStarted","Data":"5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93"}
Sep 09 21:13:16.295794 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:16.295600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:16.317278 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:16.317225    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podStartSLOduration=3.363204074 podStartE2EDuration="7.317209383s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.954005309s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:16.315737423 +0000 UTC m=+1285.876764742" watchObservedRunningTime="2026-09-09 21:13:16.317209383 +0000 UTC m=+1285.878236699"
Sep 09 21:13:17.300335 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:17.300302    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:17.301870 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:17.301842    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:13:18.304605 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:18.304570    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:13:23.309126 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:23.309097    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:13:23.309750 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:23.309721    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:13:33.310425 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:33.310388    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:13:43.310530 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:43.310486    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:13:53.309891 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:13:53.309850    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:14:03.310419 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:03.310373    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:14:13.309746 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:13.309659    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:14:23.310479 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:23.310437    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:14:33.310604 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:33.310572    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:14:39.771315 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:39.771282    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"]
Sep 09 21:14:39.771721 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:39.771553    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" containerID="cri-o://5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93" gracePeriod=30
Sep 09 21:14:39.771721 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:39.771571    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kube-rbac-proxy" containerID="cri-o://ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010" gracePeriod=30
Sep 09 21:14:40.700042 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:40.700006    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerID="ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010" exitCode=2
Sep 09 21:14:40.700213 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:40.700080    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" event={"ID":"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3","Type":"ContainerDied","Data":"ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010"}
Sep 09 21:14:43.305244 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:43.305202    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 09 21:14:43.309884 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:43.309857    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 09 21:14:44.515707 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.515680    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:14:44.588413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.588382    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-658p7\" (UniqueName: \"kubernetes.io/projected/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-api-access-658p7\") pod \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") "
Sep 09 21:14:44.588566 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.588421    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kserve-provision-location\") pod \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") "
Sep 09 21:14:44.588566 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.588443    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-rbac-proxy-sar-config\") pod \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") "
Sep 09 21:14:44.588566 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.588485    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-proxy-tls\") pod \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") "
Sep 09 21:14:44.588566 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.588500    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-cabundle-cert\") pod \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\" (UID: \"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3\") "
Sep 09 21:14:44.588892 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.588795    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" (UID: "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:14:44.588956 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.588893    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" (UID: "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:44.588998 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.588952    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" (UID: "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:44.590785 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.590746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" (UID: "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:14:44.590785 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.590742    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-api-access-658p7" (OuterVolumeSpecName: "kube-api-access-658p7") pod "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" (UID: "fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3"). InnerVolumeSpecName "kube-api-access-658p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:14:44.689522 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.689439    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-658p7\" (UniqueName: \"kubernetes.io/projected/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-api-access-658p7\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:14:44.689522 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.689467    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:14:44.689522 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.689476    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:14:44.689522 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.689486    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:14:44.689522 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.689497    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:14:44.722846 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.722815    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerID="5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93" exitCode=0
Sep 09 21:14:44.722983 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.722862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" event={"ID":"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3","Type":"ContainerDied","Data":"5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93"}
Sep 09 21:14:44.722983 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.722894    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"
Sep 09 21:14:44.722983 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.722925    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg" event={"ID":"fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3","Type":"ContainerDied","Data":"288d6f452cef7c16d8a72c51c1110669e7a8af4d088f78c37945a5aadf54969e"}
Sep 09 21:14:44.722983 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.722960    2580 scope.go:117] "RemoveContainer" containerID="ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010"
Sep 09 21:14:44.742942 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.742916    2580 scope.go:117] "RemoveContainer" containerID="5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93"
Sep 09 21:14:44.750928 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.750896    2580 scope.go:117] "RemoveContainer" containerID="d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae"
Sep 09 21:14:44.753542 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.753518    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"]
Sep 09 21:14:44.757376 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.757356    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bd8597c94-nk7fg"]
Sep 09 21:14:44.759749 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.759727    2580 scope.go:117] "RemoveContainer" containerID="ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010"
Sep 09 21:14:44.760094 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:14:44.760072    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010\": container with ID starting with ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010 not found: ID does not exist" containerID="ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010"
Sep 09 21:14:44.760152 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.760102    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010"} err="failed to get container status \"ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010\": rpc error: code = NotFound desc = could not find container \"ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010\": container with ID starting with ac980a15c4e8c40b8cf8a1bdabb2f6b8e40a3d02b6fa32d64b5816850e032010 not found: ID does not exist"
Sep 09 21:14:44.760152 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.760119    2580 scope.go:117] "RemoveContainer" containerID="5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93"
Sep 09 21:14:44.760399 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:14:44.760379    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93\": container with ID starting with 5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93 not found: ID does not exist" containerID="5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93"
Sep 09 21:14:44.760455 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.760405    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93"} err="failed to get container status \"5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93\": rpc error: code = NotFound desc = could not find container \"5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93\": container with ID starting with 5e773a7812ff857776007ae3b0213e9c54f6286099b7a95024bc28fb0dc8de93 not found: ID does not exist"
Sep 09 21:14:44.760455 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.760420    2580 scope.go:117] "RemoveContainer" containerID="d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae"
Sep 09 21:14:44.760639 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:14:44.760625    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae\": container with ID starting with d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae not found: ID does not exist" containerID="d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae"
Sep 09 21:14:44.760681 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.760641    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae"} err="failed to get container status \"d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae\": rpc error: code = NotFound desc = could not find container \"d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae\": container with ID starting with d7ba0131c0c3f41481a9089266355fc6d1b176192167db78aea8bf8e15ed7dae not found: ID does not exist"
Sep 09 21:14:44.928802 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.928750    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"]
Sep 09 21:14:44.929162 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.929148    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container"
Sep 09 21:14:44.929212 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.929164    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container"
Sep 09 21:14:44.929212 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.929175    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="storage-initializer"
Sep 09 21:14:44.929212 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.929180    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="storage-initializer"
Sep 09 21:14:44.929212 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.929194    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kube-rbac-proxy"
Sep 09 21:14:44.929212 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.929200    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kube-rbac-proxy"
Sep 09 21:14:44.929371 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.929257    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kserve-container"
Sep 09 21:14:44.929371 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.929266    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" containerName="kube-rbac-proxy"
Sep 09 21:14:44.934188 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.934165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:44.936594 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.936575    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:14:44.936722 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.936611    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 09 21:14:44.936722 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.936640    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:14:44.936964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.936647    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:14:44.936964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.936648    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 09 21:14:44.936964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.936656    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:14:44.936964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.936660    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:14:44.940381 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.940330    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3" path="/var/lib/kubelet/pods/fc8c6d21-5fc6-4e1d-b0c2-7ad7bc3541d3/volumes"
Sep 09 21:14:44.945392 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.945370    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"]
Sep 09 21:14:44.992086 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.992050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:44.992086 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.992093    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:44.992307 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.992127    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:44.992307 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.992154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:44.992307 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:44.992275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mn8t\" (UniqueName: \"kubernetes.io/projected/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-api-access-9mn8t\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.092692 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.092654    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.092692 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.092687    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.093168 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.092721    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.093168 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.092749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.093168 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.092845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mn8t\" (UniqueName: \"kubernetes.io/projected/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-api-access-9mn8t\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.093369 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.093200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.093501 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.093480    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.093501 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.093488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.095498 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.095481    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.101190 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.101172    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mn8t\" (UniqueName: \"kubernetes.io/projected/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-api-access-9mn8t\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.244935 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.244846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:14:45.522114 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.522090    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"]
Sep 09 21:14:45.524470 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:14:45.524442    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5f9bbd_83f2_4f42_8b6e_8c43c40e6dee.slice/crio-e7e1152bda39931e28c1f1a52a02fd182e000af1268d9ddfbd1b2718f48f8990 WatchSource:0}: Error finding container e7e1152bda39931e28c1f1a52a02fd182e000af1268d9ddfbd1b2718f48f8990: Status 404 returned error can't find the container with id e7e1152bda39931e28c1f1a52a02fd182e000af1268d9ddfbd1b2718f48f8990
Sep 09 21:14:45.730505 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.730469    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" event={"ID":"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee","Type":"ContainerStarted","Data":"5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89"}
Sep 09 21:14:45.730688 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:45.730542    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" event={"ID":"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee","Type":"ContainerStarted","Data":"e7e1152bda39931e28c1f1a52a02fd182e000af1268d9ddfbd1b2718f48f8990"}
Sep 09 21:14:49.752953 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:49.752862    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerID="5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89" exitCode=0
Sep 09 21:14:49.752953 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:49.752941    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" event={"ID":"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee","Type":"ContainerDied","Data":"5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89"}
Sep 09 21:14:49.754149 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:14:49.754126    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:17:05.003685 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:05.003657    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:17:05.014153 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:05.003658    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:17:06.515855 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:06.515816    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" event={"ID":"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee","Type":"ContainerStarted","Data":"639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f"}
Sep 09 21:17:06.516250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:06.515875    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" event={"ID":"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee","Type":"ContainerStarted","Data":"39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42"}
Sep 09 21:17:06.516250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:06.515907    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:17:06.538999 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:06.538954    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" podStartSLOduration=2.814177398 podStartE2EDuration="2m22.538938253s" totalImagesPullingTime="2m15.836919195s" totalInitContainerRuntime="3.88784166s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:17:06.536688036 +0000 UTC m=+1516.097715392" watchObservedRunningTime="2026-09-09 21:17:06.538938253 +0000 UTC m=+1516.099965569"
Sep 09 21:17:07.521252 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:07.521220    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:17:13.531652 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:13.531576    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:17:38.904454 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:38.904420    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"]
Sep 09 21:17:38.945057 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:38.945022    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"]
Sep 09 21:17:38.945236 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:38.945139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:38.947498 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:38.947474    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 09 21:17:38.947738 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:38.947723    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 09 21:17:39.059202 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.059161    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.059371 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.059208    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr9n9\" (UniqueName: \"kubernetes.io/projected/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-api-access-cr9n9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.059371 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.059267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.059371 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.059309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.059550 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.059418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.160456 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.160370    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.160456 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.160410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr9n9\" (UniqueName: \"kubernetes.io/projected/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-api-access-cr9n9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.160456 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.160438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.160741 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.160470    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.160741 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.160548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.160741 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:17:39.160583    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-serving-cert: secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 09 21:17:39.160741 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:17:39.160661    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-proxy-tls podName:a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad nodeName:}" failed. No retries permitted until 2026-09-09 21:17:39.660639993 +0000 UTC m=+1549.221667293 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-proxy-tls") pod "isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" (UID: "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad") : secret "isvc-autogluon-ts-v1-noslash-predictor-serving-cert" not found
Sep 09 21:17:39.161189 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.161162    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.161263 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.161210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.161263 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.161217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.169432 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.169405    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr9n9\" (UniqueName: \"kubernetes.io/projected/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-api-access-cr9n9\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.664599 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.664566    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.667149 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.667131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:39.856950 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:39.856909    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:17:40.281983 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:40.281958    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"]
Sep 09 21:17:40.284571 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:17:40.284543    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ac9fc3_0ac5_4a37_9e65_623a0b02bbad.slice/crio-26da57f08dd6ba4accd3916c9514498a91b68abb8f14ba6fc75cb46643635164 WatchSource:0}: Error finding container 26da57f08dd6ba4accd3916c9514498a91b68abb8f14ba6fc75cb46643635164: Status 404 returned error can't find the container with id 26da57f08dd6ba4accd3916c9514498a91b68abb8f14ba6fc75cb46643635164
Sep 09 21:17:40.679656 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:40.679623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" event={"ID":"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad","Type":"ContainerStarted","Data":"cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd"}
Sep 09 21:17:40.679855 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:40.679682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" event={"ID":"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad","Type":"ContainerStarted","Data":"26da57f08dd6ba4accd3916c9514498a91b68abb8f14ba6fc75cb46643635164"}
Sep 09 21:17:43.552160 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:43.552113    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 21:17:49.724728 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:49.724696    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerID="cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd" exitCode=0
Sep 09 21:17:49.725172 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:49.724792    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" event={"ID":"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad","Type":"ContainerDied","Data":"cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd"}
Sep 09 21:17:53.534751 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:53.534721    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:17:55.004326 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:55.004291    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"]
Sep 09 21:17:55.004814 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:55.004604    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kserve-container" containerID="cri-o://39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42" gracePeriod=30
Sep 09 21:17:55.004814 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:55.004674    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kube-rbac-proxy" containerID="cri-o://639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f" gracePeriod=30
Sep 09 21:17:55.770469 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:55.770438    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerID="639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f" exitCode=2
Sep 09 21:17:55.770645 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:55.770493    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" event={"ID":"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee","Type":"ContainerDied","Data":"639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f"}
Sep 09 21:17:58.526307 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:17:58.526263    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 09 21:18:00.190930 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.190890    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"]
Sep 09 21:18:00.194361 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.194338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.197502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.197476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 09 21:18:00.197772 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.197743    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:18:00.206993 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.206972    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"]
Sep 09 21:18:00.249304 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.249279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.249433 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.249349    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.249433 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.249382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.249433 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.249403    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v74m2\" (UniqueName: \"kubernetes.io/projected/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-api-access-v74m2\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.249433 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.249427    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.349995 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.349962    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.350180 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.350049    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.350180 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.350133    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.350180 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.350170    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.350352 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.350204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v74m2\" (UniqueName: \"kubernetes.io/projected/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-api-access-v74m2\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.350352 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:18:00.350277    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 09 21:18:00.350451 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:18:00.350372    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-proxy-tls podName:6c421d3c-86e4-4db6-a723-ef73d4a3be9f nodeName:}" failed. No retries permitted until 2026-09-09 21:18:00.850349186 +0000 UTC m=+1570.411376498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-proxy-tls") pod "isvc-sklearn-runtime-predictor-6446f4d54b-npc84" (UID: "6c421d3c-86e4-4db6-a723-ef73d4a3be9f") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Sep 09 21:18:00.350833 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.350806    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.350963 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.350883    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.350963 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.350910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.361604 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.361554    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v74m2\" (UniqueName: \"kubernetes.io/projected/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-api-access-v74m2\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.855008 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.854930    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:00.858010 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:00.857980    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-6446f4d54b-npc84\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:01.107868 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:01.107789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:01.423955 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:01.423930    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"]
Sep 09 21:18:01.425748 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:18:01.425726    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c421d3c_86e4_4db6_a723_ef73d4a3be9f.slice/crio-4367ec7892e248d6a47e6fd605dfc82cb02e2d8d8dbbe14c365c39b4f4396894 WatchSource:0}: Error finding container 4367ec7892e248d6a47e6fd605dfc82cb02e2d8d8dbbe14c365c39b4f4396894: Status 404 returned error can't find the container with id 4367ec7892e248d6a47e6fd605dfc82cb02e2d8d8dbbe14c365c39b4f4396894
Sep 09 21:18:01.809635 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:01.809548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" event={"ID":"6c421d3c-86e4-4db6-a723-ef73d4a3be9f","Type":"ContainerStarted","Data":"2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4"}
Sep 09 21:18:01.809635 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:01.809625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" event={"ID":"6c421d3c-86e4-4db6-a723-ef73d4a3be9f","Type":"ContainerStarted","Data":"4367ec7892e248d6a47e6fd605dfc82cb02e2d8d8dbbe14c365c39b4f4396894"}
Sep 09 21:18:03.272987 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.272945    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:18:03.376721 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.376686    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kserve-provision-location\") pod \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") "
Sep 09 21:18:03.376939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.376731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-cabundle-cert\") pod \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") "
Sep 09 21:18:03.376939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.376825    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mn8t\" (UniqueName: \"kubernetes.io/projected/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-api-access-9mn8t\") pod \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") "
Sep 09 21:18:03.376939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.376872    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-proxy-tls\") pod \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") "
Sep 09 21:18:03.376939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.376937    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-rbac-proxy-sar-config\") pod \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\" (UID: \"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee\") "
Sep 09 21:18:03.377171 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.377088    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" (UID: "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:18:03.377235 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.377191    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:03.377291 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.377270    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" (UID: "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:18:03.377576 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.377551    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" (UID: "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:18:03.379485 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.379457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-api-access-9mn8t" (OuterVolumeSpecName: "kube-api-access-9mn8t") pod "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" (UID: "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee"). InnerVolumeSpecName "kube-api-access-9mn8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:18:03.379581 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.379496    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" (UID: "4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:18:03.478146 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.478112    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mn8t\" (UniqueName: \"kubernetes.io/projected/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-api-access-9mn8t\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:03.478146 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.478145    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:03.478146 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.478156    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:03.478454 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.478167    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:03.823203 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.823069    2580 generic.go:356] "Generic (PLEG): container finished" podID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerID="39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42" exitCode=0
Sep 09 21:18:03.823203 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.823112    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" event={"ID":"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee","Type":"ContainerDied","Data":"39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42"}
Sep 09 21:18:03.823203 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.823135    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"
Sep 09 21:18:03.823203 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.823193    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq" event={"ID":"4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee","Type":"ContainerDied","Data":"e7e1152bda39931e28c1f1a52a02fd182e000af1268d9ddfbd1b2718f48f8990"}
Sep 09 21:18:03.823624 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.823248    2580 scope.go:117] "RemoveContainer" containerID="639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f"
Sep 09 21:18:03.834577 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.834559    2580 scope.go:117] "RemoveContainer" containerID="39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42"
Sep 09 21:18:03.845149 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.845117    2580 scope.go:117] "RemoveContainer" containerID="5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89"
Sep 09 21:18:03.847404 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.847377    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"]
Sep 09 21:18:03.851949 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.851926    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-2fhwq"]
Sep 09 21:18:03.856100 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.856067    2580 scope.go:117] "RemoveContainer" containerID="639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f"
Sep 09 21:18:03.856432 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:18:03.856406    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f\": container with ID starting with 639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f not found: ID does not exist" containerID="639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f"
Sep 09 21:18:03.856503 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.856443    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f"} err="failed to get container status \"639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f\": rpc error: code = NotFound desc = could not find container \"639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f\": container with ID starting with 639035551d1ac5e77525c7fced104cf9bf5699d8c25eeeccc41debf6fa53ee7f not found: ID does not exist"
Sep 09 21:18:03.856503 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.856479    2580 scope.go:117] "RemoveContainer" containerID="39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42"
Sep 09 21:18:03.856842 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:18:03.856811    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42\": container with ID starting with 39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42 not found: ID does not exist" containerID="39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42"
Sep 09 21:18:03.856949 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.856846    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42"} err="failed to get container status \"39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42\": rpc error: code = NotFound desc = could not find container \"39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42\": container with ID starting with 39609d7691133fcb39e43755175426fbe2788b83ce7b5577cc80460f7517bf42 not found: ID does not exist"
Sep 09 21:18:03.856949 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.856864    2580 scope.go:117] "RemoveContainer" containerID="5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89"
Sep 09 21:18:03.857184 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:18:03.857160    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89\": container with ID starting with 5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89 not found: ID does not exist" containerID="5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89"
Sep 09 21:18:03.857237 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:03.857190    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89"} err="failed to get container status \"5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89\": rpc error: code = NotFound desc = could not find container \"5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89\": container with ID starting with 5f07e7299601f1e23df6141e73615f50c72d98c537e9ae5c96be7ec6d2554f89 not found: ID does not exist"
Sep 09 21:18:04.937459 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:04.937424    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" path="/var/lib/kubelet/pods/4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee/volumes"
Sep 09 21:18:08.856845 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:08.856808    2580 generic.go:356] "Generic (PLEG): container finished" podID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerID="2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4" exitCode=0
Sep 09 21:18:08.857345 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:08.856884    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" event={"ID":"6c421d3c-86e4-4db6-a723-ef73d4a3be9f","Type":"ContainerDied","Data":"2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4"}
Sep 09 21:18:09.866639 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:09.866604    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" event={"ID":"6c421d3c-86e4-4db6-a723-ef73d4a3be9f","Type":"ContainerStarted","Data":"4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233"}
Sep 09 21:18:09.867219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:09.866673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" event={"ID":"6c421d3c-86e4-4db6-a723-ef73d4a3be9f","Type":"ContainerStarted","Data":"237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526"}
Sep 09 21:18:09.867219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:09.867154    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:09.867219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:09.867216    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:09.868848 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:09.868816    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 21:18:09.887435 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:09.887372    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" podStartSLOduration=2.944200757 podStartE2EDuration="9.88735234s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.943151583s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:09.886932324 +0000 UTC m=+1579.447959656" watchObservedRunningTime="2026-09-09 21:18:09.88735234 +0000 UTC m=+1579.448379662"
Sep 09 21:18:10.871776 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:10.871725    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 21:18:15.877785 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:15.877739    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:15.878294 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:15.878259    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 21:18:25.879087 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:25.879046    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 09 21:18:35.879711 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:35.879670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:42.372852 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:42.372814    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"]
Sep 09 21:18:42.373411 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:42.373225    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kserve-container" containerID="cri-o://237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526" gracePeriod=30
Sep 09 21:18:42.373411 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:42.373322    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kube-rbac-proxy" containerID="cri-o://4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233" gracePeriod=30
Sep 09 21:18:43.080314 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.080282    2580 generic.go:356] "Generic (PLEG): container finished" podID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerID="4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233" exitCode=2
Sep 09 21:18:43.080506 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.080353    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" event={"ID":"6c421d3c-86e4-4db6-a723-ef73d4a3be9f","Type":"ContainerDied","Data":"4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233"}
Sep 09 21:18:43.460350 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.460327    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:43.548852 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.548822    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kserve-provision-location\") pod \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") "
Sep 09 21:18:43.549011 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.548885    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-cabundle-cert\") pod \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") "
Sep 09 21:18:43.549011 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.548916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-rbac-proxy-sar-config\") pod \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") "
Sep 09 21:18:43.549011 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.548949    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-proxy-tls\") pod \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") "
Sep 09 21:18:43.549207 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.549159    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v74m2\" (UniqueName: \"kubernetes.io/projected/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-api-access-v74m2\") pod \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\" (UID: \"6c421d3c-86e4-4db6-a723-ef73d4a3be9f\") "
Sep 09 21:18:43.549374 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.549340    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c421d3c-86e4-4db6-a723-ef73d4a3be9f" (UID: "6c421d3c-86e4-4db6-a723-ef73d4a3be9f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:18:43.549374 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.549346    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c421d3c-86e4-4db6-a723-ef73d4a3be9f" (UID: "6c421d3c-86e4-4db6-a723-ef73d4a3be9f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:18:43.549573 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.549513    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:43.549573 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.549533    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:43.551816 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.551789    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c421d3c-86e4-4db6-a723-ef73d4a3be9f" (UID: "6c421d3c-86e4-4db6-a723-ef73d4a3be9f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:18:43.551938 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.551885    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-api-access-v74m2" (OuterVolumeSpecName: "kube-api-access-v74m2") pod "6c421d3c-86e4-4db6-a723-ef73d4a3be9f" (UID: "6c421d3c-86e4-4db6-a723-ef73d4a3be9f"). InnerVolumeSpecName "kube-api-access-v74m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:18:43.575064 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.575040    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c421d3c-86e4-4db6-a723-ef73d4a3be9f" (UID: "6c421d3c-86e4-4db6-a723-ef73d4a3be9f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:18:43.650584 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.650500    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:43.650584 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.650536    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v74m2\" (UniqueName: \"kubernetes.io/projected/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kube-api-access-v74m2\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:43.650584 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:43.650553    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c421d3c-86e4-4db6-a723-ef73d4a3be9f-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:18:44.089528 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.089442    2580 generic.go:356] "Generic (PLEG): container finished" podID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerID="237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526" exitCode=0
Sep 09 21:18:44.089528 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.089508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" event={"ID":"6c421d3c-86e4-4db6-a723-ef73d4a3be9f","Type":"ContainerDied","Data":"237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526"}
Sep 09 21:18:44.089728 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.089561    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"
Sep 09 21:18:44.089728 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.089573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84" event={"ID":"6c421d3c-86e4-4db6-a723-ef73d4a3be9f","Type":"ContainerDied","Data":"4367ec7892e248d6a47e6fd605dfc82cb02e2d8d8dbbe14c365c39b4f4396894"}
Sep 09 21:18:44.089728 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.089618    2580 scope.go:117] "RemoveContainer" containerID="4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233"
Sep 09 21:18:44.102785 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.102733    2580 scope.go:117] "RemoveContainer" containerID="237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526"
Sep 09 21:18:44.113871 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.113621    2580 scope.go:117] "RemoveContainer" containerID="2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4"
Sep 09 21:18:44.116678 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.116653    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"]
Sep 09 21:18:44.119718 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.119698    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-6446f4d54b-npc84"]
Sep 09 21:18:44.124013 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.123977    2580 scope.go:117] "RemoveContainer" containerID="4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233"
Sep 09 21:18:44.124362 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:18:44.124333    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233\": container with ID starting with 4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233 not found: ID does not exist" containerID="4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233"
Sep 09 21:18:44.124440 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.124375    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233"} err="failed to get container status \"4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233\": rpc error: code = NotFound desc = could not find container \"4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233\": container with ID starting with 4d64b4a5c0dbae12fe880916f681aac18b446215878432ec0068892b04f37233 not found: ID does not exist"
Sep 09 21:18:44.124440 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.124403    2580 scope.go:117] "RemoveContainer" containerID="237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526"
Sep 09 21:18:44.124736 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:18:44.124706    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526\": container with ID starting with 237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526 not found: ID does not exist" containerID="237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526"
Sep 09 21:18:44.124873 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.124743    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526"} err="failed to get container status \"237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526\": rpc error: code = NotFound desc = could not find container \"237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526\": container with ID starting with 237d9d9387d2689d702faec598727c97e777327f39053ccea041c008f486a526 not found: ID does not exist"
Sep 09 21:18:44.124873 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.124784    2580 scope.go:117] "RemoveContainer" containerID="2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4"
Sep 09 21:18:44.125104 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:18:44.125080    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4\": container with ID starting with 2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4 not found: ID does not exist" containerID="2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4"
Sep 09 21:18:44.125219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.125199    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4"} err="failed to get container status \"2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4\": rpc error: code = NotFound desc = could not find container \"2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4\": container with ID starting with 2f8ed519482c55e4e047e4d9b0051d3dd1e7a409f2aa87e42baa46b4f8fa31e4 not found: ID does not exist"
Sep 09 21:18:44.936183 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:44.936151    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" path="/var/lib/kubelet/pods/6c421d3c-86e4-4db6-a723-ef73d4a3be9f/volumes"
Sep 09 21:18:47.622320 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622286    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"]
Sep 09 21:18:47.622859 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622840    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kube-rbac-proxy"
Sep 09 21:18:47.622939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622862    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kube-rbac-proxy"
Sep 09 21:18:47.622939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622876    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="storage-initializer"
Sep 09 21:18:47.622939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622885    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="storage-initializer"
Sep 09 21:18:47.622939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622902    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kube-rbac-proxy"
Sep 09 21:18:47.622939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622911    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kube-rbac-proxy"
Sep 09 21:18:47.622939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622940    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kserve-container"
Sep 09 21:18:47.623250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622948    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kserve-container"
Sep 09 21:18:47.623250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622960    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="storage-initializer"
Sep 09 21:18:47.623250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622968    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="storage-initializer"
Sep 09 21:18:47.623250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622977    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kserve-container"
Sep 09 21:18:47.623250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.622985    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kserve-container"
Sep 09 21:18:47.623250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.623075    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kube-rbac-proxy"
Sep 09 21:18:47.623250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.623089    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kserve-container"
Sep 09 21:18:47.623250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.623099    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c421d3c-86e4-4db6-a723-ef73d4a3be9f" containerName="kube-rbac-proxy"
Sep 09 21:18:47.623250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.623108    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a5f9bbd-83f2-4f42-8b6e-8c43c40e6dee" containerName="kserve-container"
Sep 09 21:18:47.626712 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.626688    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.629992 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.629964    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 09 21:18:47.630125 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.630047    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:18:47.644094 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.644070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"]
Sep 09 21:18:47.791217 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.791179    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9qk\" (UniqueName: \"kubernetes.io/projected/dad1f544-56aa-4039-be5a-117fc2223507-kube-api-access-mc9qk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.791426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.791233    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.791426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.791301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.791426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.791334    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dad1f544-56aa-4039-be5a-117fc2223507-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.791426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.791366    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dad1f544-56aa-4039-be5a-117fc2223507-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.892404 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.892315    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9qk\" (UniqueName: \"kubernetes.io/projected/dad1f544-56aa-4039-be5a-117fc2223507-kube-api-access-mc9qk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.892404 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.892372    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.892682 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.892413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.892682 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.892448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dad1f544-56aa-4039-be5a-117fc2223507-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.892682 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.892477    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dad1f544-56aa-4039-be5a-117fc2223507-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.892969 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.892947    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dad1f544-56aa-4039-be5a-117fc2223507-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.893251 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.893222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.893251 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.893238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.895531 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.895502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dad1f544-56aa-4039-be5a-117fc2223507-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.902799 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.902736    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9qk\" (UniqueName: \"kubernetes.io/projected/dad1f544-56aa-4039-be5a-117fc2223507-kube-api-access-mc9qk\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:47.939290 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:47.939249    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:48.250078 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:48.249095    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"]
Sep 09 21:18:49.119655 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:49.119609    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" event={"ID":"dad1f544-56aa-4039-be5a-117fc2223507","Type":"ContainerStarted","Data":"5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4"}
Sep 09 21:18:49.120133 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:49.119682    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" event={"ID":"dad1f544-56aa-4039-be5a-117fc2223507","Type":"ContainerStarted","Data":"6394a4f90e038e6a8fb68ca0a01d383ea196562e5c69d97ef2eb8519fe0937c5"}
Sep 09 21:18:53.144910 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:53.144871    2580 generic.go:356] "Generic (PLEG): container finished" podID="dad1f544-56aa-4039-be5a-117fc2223507" containerID="5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4" exitCode=0
Sep 09 21:18:53.145392 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:53.144946    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" event={"ID":"dad1f544-56aa-4039-be5a-117fc2223507","Type":"ContainerDied","Data":"5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4"}
Sep 09 21:18:54.154879 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:54.154832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" event={"ID":"dad1f544-56aa-4039-be5a-117fc2223507","Type":"ContainerStarted","Data":"803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b"}
Sep 09 21:18:54.155357 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:54.154958    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" event={"ID":"dad1f544-56aa-4039-be5a-117fc2223507","Type":"ContainerStarted","Data":"c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e"}
Sep 09 21:18:54.155357 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:54.155323    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:54.155490 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:54.155400    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:18:54.175622 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:18:54.175576    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" podStartSLOduration=3.173261647 podStartE2EDuration="7.175558914s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.002297267s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:54.173305795 +0000 UTC m=+1623.734333136" watchObservedRunningTime="2026-09-09 21:18:54.175558914 +0000 UTC m=+1623.736586232"
Sep 09 21:19:00.167694 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:19:00.167661    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:19:30.253389 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:19:30.253339    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 21:19:40.252148 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:19:40.252094    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 21:19:50.171304 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:19:50.171270    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:19:57.625777 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:19:57.625721    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"]
Sep 09 21:19:57.626537 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:19:57.626059    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kserve-container" containerID="cri-o://c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e" gracePeriod=30
Sep 09 21:19:57.626537 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:19:57.626310    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kube-rbac-proxy" containerID="cri-o://803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b" gracePeriod=30
Sep 09 21:20:00.162127 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:00.162083    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 09 21:20:01.209994 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:01.209939    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.50:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:20:02.547574 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.547493    2580 generic.go:356] "Generic (PLEG): container finished" podID="dad1f544-56aa-4039-be5a-117fc2223507" containerID="803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b" exitCode=2
Sep 09 21:20:02.547950 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.547573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" event={"ID":"dad1f544-56aa-4039-be5a-117fc2223507","Type":"ContainerDied","Data":"803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b"}
Sep 09 21:20:02.832307 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.832129    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"]
Sep 09 21:20:02.843961 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.843917    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:02.848644 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.848621    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 09 21:20:02.848644 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.848634    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 09 21:20:02.866518 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.866489    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"]
Sep 09 21:20:02.994899 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.994868    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dd50125-8daa-420c-8182-34efc7182614-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:02.995069 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.994916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:02.995069 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.994999    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dstql\" (UniqueName: \"kubernetes.io/projected/0dd50125-8daa-420c-8182-34efc7182614-kube-api-access-dstql\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:02.995069 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.995038    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:02.995069 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:02.995069    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd50125-8daa-420c-8182-34efc7182614-proxy-tls\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.096532 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.096445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstql\" (UniqueName: \"kubernetes.io/projected/0dd50125-8daa-420c-8182-34efc7182614-kube-api-access-dstql\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.096532 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.096486    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.096811 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.096664    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd50125-8daa-420c-8182-34efc7182614-proxy-tls\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.096811 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.096723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dd50125-8daa-420c-8182-34efc7182614-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.096811 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.096754    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.097193 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.097170    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dd50125-8daa-420c-8182-34efc7182614-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.097294 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.097205    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.097447 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.097321    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.099387 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.099359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd50125-8daa-420c-8182-34efc7182614-proxy-tls\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.108220 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.108193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstql\" (UniqueName: \"kubernetes.io/projected/0dd50125-8daa-420c-8182-34efc7182614-kube-api-access-dstql\") pod \"isvc-sklearn-v2-predictor-64bfb47598-k2nn8\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.154155 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.154117    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:03.455786 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.455691    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"]
Sep 09 21:20:03.459511 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:20:03.459480    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd50125_8daa_420c_8182_34efc7182614.slice/crio-b8e0f35cc4669fc07226b99b5c0ccc882f83a54e92ed49071c8f76a034a6252e WatchSource:0}: Error finding container b8e0f35cc4669fc07226b99b5c0ccc882f83a54e92ed49071c8f76a034a6252e: Status 404 returned error can't find the container with id b8e0f35cc4669fc07226b99b5c0ccc882f83a54e92ed49071c8f76a034a6252e
Sep 09 21:20:03.555732 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:03.555703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" event={"ID":"0dd50125-8daa-420c-8182-34efc7182614","Type":"ContainerStarted","Data":"b8e0f35cc4669fc07226b99b5c0ccc882f83a54e92ed49071c8f76a034a6252e"}
Sep 09 21:20:04.564990 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:04.564950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" event={"ID":"0dd50125-8daa-420c-8182-34efc7182614","Type":"ContainerStarted","Data":"63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741"}
Sep 09 21:20:04.567536 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:04.567506    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" event={"ID":"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad","Type":"ContainerStarted","Data":"d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c"}
Sep 09 21:20:04.567688 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:04.567564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" event={"ID":"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad","Type":"ContainerStarted","Data":"0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2"}
Sep 09 21:20:04.567688 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:04.567606    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:20:04.611114 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:04.611064    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" podStartSLOduration=4.109640318 podStartE2EDuration="2m26.611046942s" totalImagesPullingTime="2m13.753915552s" totalInitContainerRuntime="8.747491072s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:04.608099243 +0000 UTC m=+1694.169126582" watchObservedRunningTime="2026-09-09 21:20:04.611046942 +0000 UTC m=+1694.172074258"
Sep 09 21:20:05.162103 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.162062    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 09 21:20:05.574714 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.574048    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:20:05.575381 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.575343    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 21:20:05.808814 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.808732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:20:05.921639 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.921603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dad1f544-56aa-4039-be5a-117fc2223507-kserve-provision-location\") pod \"dad1f544-56aa-4039-be5a-117fc2223507\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") "
Sep 09 21:20:05.921910 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.921670    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-kube-rbac-proxy-sar-config\") pod \"dad1f544-56aa-4039-be5a-117fc2223507\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") "
Sep 09 21:20:05.921910 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.921710    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-cabundle-cert\") pod \"dad1f544-56aa-4039-be5a-117fc2223507\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") "
Sep 09 21:20:05.921910 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.921821    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dad1f544-56aa-4039-be5a-117fc2223507-proxy-tls\") pod \"dad1f544-56aa-4039-be5a-117fc2223507\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") "
Sep 09 21:20:05.921910 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.921887    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mc9qk\" (UniqueName: \"kubernetes.io/projected/dad1f544-56aa-4039-be5a-117fc2223507-kube-api-access-mc9qk\") pod \"dad1f544-56aa-4039-be5a-117fc2223507\" (UID: \"dad1f544-56aa-4039-be5a-117fc2223507\") "
Sep 09 21:20:05.922134 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.922018    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dad1f544-56aa-4039-be5a-117fc2223507-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dad1f544-56aa-4039-be5a-117fc2223507" (UID: "dad1f544-56aa-4039-be5a-117fc2223507"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:05.922134 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.922015    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dad1f544-56aa-4039-be5a-117fc2223507" (UID: "dad1f544-56aa-4039-be5a-117fc2223507"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:05.922134 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.922054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dad1f544-56aa-4039-be5a-117fc2223507" (UID: "dad1f544-56aa-4039-be5a-117fc2223507"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:05.922234 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.922184    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:05.922234 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.922197    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dad1f544-56aa-4039-be5a-117fc2223507-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:05.922234 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.922206    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dad1f544-56aa-4039-be5a-117fc2223507-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:05.924147 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.924121    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dad1f544-56aa-4039-be5a-117fc2223507-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dad1f544-56aa-4039-be5a-117fc2223507" (UID: "dad1f544-56aa-4039-be5a-117fc2223507"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:20:05.924286 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:05.924250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad1f544-56aa-4039-be5a-117fc2223507-kube-api-access-mc9qk" (OuterVolumeSpecName: "kube-api-access-mc9qk") pod "dad1f544-56aa-4039-be5a-117fc2223507" (UID: "dad1f544-56aa-4039-be5a-117fc2223507"). InnerVolumeSpecName "kube-api-access-mc9qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:06.022701 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.022663    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dad1f544-56aa-4039-be5a-117fc2223507-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:06.022701 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.022689    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9qk\" (UniqueName: \"kubernetes.io/projected/dad1f544-56aa-4039-be5a-117fc2223507-kube-api-access-mc9qk\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:06.581507 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.581476    2580 generic.go:356] "Generic (PLEG): container finished" podID="dad1f544-56aa-4039-be5a-117fc2223507" containerID="c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e" exitCode=0
Sep 09 21:20:06.582044 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.581548    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"
Sep 09 21:20:06.582044 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.581568    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" event={"ID":"dad1f544-56aa-4039-be5a-117fc2223507","Type":"ContainerDied","Data":"c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e"}
Sep 09 21:20:06.582044 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.581635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv" event={"ID":"dad1f544-56aa-4039-be5a-117fc2223507","Type":"ContainerDied","Data":"6394a4f90e038e6a8fb68ca0a01d383ea196562e5c69d97ef2eb8519fe0937c5"}
Sep 09 21:20:06.582044 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.581675    2580 scope.go:117] "RemoveContainer" containerID="803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b"
Sep 09 21:20:06.582318 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.582288    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 21:20:06.594995 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.593472    2580 scope.go:117] "RemoveContainer" containerID="c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e"
Sep 09 21:20:06.603222 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.603202    2580 scope.go:117] "RemoveContainer" containerID="5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4"
Sep 09 21:20:06.612851 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.612823    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"]
Sep 09 21:20:06.613175 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.613159    2580 scope.go:117] "RemoveContainer" containerID="803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b"
Sep 09 21:20:06.613509 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:20:06.613488    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b\": container with ID starting with 803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b not found: ID does not exist" containerID="803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b"
Sep 09 21:20:06.613579 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.613516    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b"} err="failed to get container status \"803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b\": rpc error: code = NotFound desc = could not find container \"803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b\": container with ID starting with 803f61df23dfd89c2c32005e661f627f2c5e6f7c127e1abfdd47b528f536409b not found: ID does not exist"
Sep 09 21:20:06.613579 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.613534    2580 scope.go:117] "RemoveContainer" containerID="c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e"
Sep 09 21:20:06.613842 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:20:06.613816    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e\": container with ID starting with c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e not found: ID does not exist" containerID="c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e"
Sep 09 21:20:06.613924 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.613854    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e"} err="failed to get container status \"c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e\": rpc error: code = NotFound desc = could not find container \"c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e\": container with ID starting with c37fb0d4a3c6284139a799a1f5f29cfc484061e558373f8b7bd766aa5829100e not found: ID does not exist"
Sep 09 21:20:06.613924 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.613878    2580 scope.go:117] "RemoveContainer" containerID="5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4"
Sep 09 21:20:06.614138 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:20:06.614122    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4\": container with ID starting with 5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4 not found: ID does not exist" containerID="5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4"
Sep 09 21:20:06.614186 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.614143    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4"} err="failed to get container status \"5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4\": rpc error: code = NotFound desc = could not find container \"5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4\": container with ID starting with 5474bec2950065ea9da87d6a740ccae0fa921c8864535ba0d5638b7b304736e4 not found: ID does not exist"
Sep 09 21:20:06.621573 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.621548    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-4vhbv"]
Sep 09 21:20:06.935659 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:06.935624    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dad1f544-56aa-4039-be5a-117fc2223507" path="/var/lib/kubelet/pods/dad1f544-56aa-4039-be5a-117fc2223507/volumes"
Sep 09 21:20:08.596870 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:08.596839    2580 generic.go:356] "Generic (PLEG): container finished" podID="0dd50125-8daa-420c-8182-34efc7182614" containerID="63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741" exitCode=0
Sep 09 21:20:08.597263 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:08.596923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" event={"ID":"0dd50125-8daa-420c-8182-34efc7182614","Type":"ContainerDied","Data":"63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741"}
Sep 09 21:20:09.604658 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:09.604623    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" event={"ID":"0dd50125-8daa-420c-8182-34efc7182614","Type":"ContainerStarted","Data":"ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65"}
Sep 09 21:20:09.605151 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:09.604695    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" event={"ID":"0dd50125-8daa-420c-8182-34efc7182614","Type":"ContainerStarted","Data":"b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42"}
Sep 09 21:20:09.605151 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:09.604931    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:09.627356 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:09.627292    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podStartSLOduration=3.273520522 podStartE2EDuration="7.627272794s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.353752272s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:09.625460094 +0000 UTC m=+1699.186487410" watchObservedRunningTime="2026-09-09 21:20:09.627272794 +0000 UTC m=+1699.188300116"
Sep 09 21:20:10.610208 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:10.610174    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:10.611964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:10.611927    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:20:11.588520 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:11.588438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:20:11.589097 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:11.589073    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 09 21:20:11.615445 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:11.615407    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:20:16.621066 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:16.621030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:20:16.621685 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:16.621657    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:20:21.589937 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:21.589906    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:20:26.622471 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:26.622428    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:20:34.230503 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:34.230464    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"]
Sep 09 21:20:34.231015 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:34.230797    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kserve-container" containerID="cri-o://0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2" gracePeriod=30
Sep 09 21:20:34.231015 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:34.230835    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kube-rbac-proxy" containerID="cri-o://d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c" gracePeriod=30
Sep 09 21:20:34.729683 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:34.729650    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerID="d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c" exitCode=2
Sep 09 21:20:34.729877 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:34.729726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" event={"ID":"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad","Type":"ContainerDied","Data":"d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c"}
Sep 09 21:20:35.617173 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.617140    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:20:35.663801 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.663702    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-cabundle-cert\") pod \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") "
Sep 09 21:20:35.663801 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.663791    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-proxy-tls\") pod \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") "
Sep 09 21:20:35.664030 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.663903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kserve-provision-location\") pod \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") "
Sep 09 21:20:35.664030 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.663965    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr9n9\" (UniqueName: \"kubernetes.io/projected/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-api-access-cr9n9\") pod \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") "
Sep 09 21:20:35.664030 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.664019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-rbac-proxy-sar-config\") pod \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\" (UID: \"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad\") "
Sep 09 21:20:35.664201 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.664144    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" (UID: "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:35.664318 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.664294    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:35.664447 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.664369    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" (UID: "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:20:35.664853 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.664829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" (UID: "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:20:35.665994 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.665975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" (UID: "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:20:35.666104 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.666085    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-api-access-cr9n9" (OuterVolumeSpecName: "kube-api-access-cr9n9") pod "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" (UID: "a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad"). InnerVolumeSpecName "kube-api-access-cr9n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:20:35.736724 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.736691    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerID="0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2" exitCode=0
Sep 09 21:20:35.736937 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.736745    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"
Sep 09 21:20:35.736937 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.736729    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" event={"ID":"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad","Type":"ContainerDied","Data":"0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2"}
Sep 09 21:20:35.736937 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.736831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6" event={"ID":"a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad","Type":"ContainerDied","Data":"26da57f08dd6ba4accd3916c9514498a91b68abb8f14ba6fc75cb46643635164"}
Sep 09 21:20:35.736937 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.736861    2580 scope.go:117] "RemoveContainer" containerID="d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c"
Sep 09 21:20:35.747009 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.746992    2580 scope.go:117] "RemoveContainer" containerID="0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2"
Sep 09 21:20:35.756679 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.756656    2580 scope.go:117] "RemoveContainer" containerID="cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd"
Sep 09 21:20:35.764659 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.764636    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:35.764780 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.764662    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:35.764780 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.764676    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr9n9\" (UniqueName: \"kubernetes.io/projected/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-api-access-cr9n9\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:35.764780 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.764691    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:20:35.765526 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.765496    2580 scope.go:117] "RemoveContainer" containerID="d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c"
Sep 09 21:20:35.765842 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:20:35.765821    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c\": container with ID starting with d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c not found: ID does not exist" containerID="d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c"
Sep 09 21:20:35.765888 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.765852    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c"} err="failed to get container status \"d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c\": rpc error: code = NotFound desc = could not find container \"d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c\": container with ID starting with d272bd1ef184865705e5cab42dcdcd6c501dc09568ee638e469ec51ec021af2c not found: ID does not exist"
Sep 09 21:20:35.765888 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.765870    2580 scope.go:117] "RemoveContainer" containerID="0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2"
Sep 09 21:20:35.766127 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:20:35.766106    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2\": container with ID starting with 0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2 not found: ID does not exist" containerID="0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2"
Sep 09 21:20:35.766207 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.766137    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2"} err="failed to get container status \"0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2\": rpc error: code = NotFound desc = could not find container \"0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2\": container with ID starting with 0b911a97add51ba3e22f99d331c57d8e806ad0e94d10bd3de54b8d6675b415c2 not found: ID does not exist"
Sep 09 21:20:35.766207 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.766159    2580 scope.go:117] "RemoveContainer" containerID="cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd"
Sep 09 21:20:35.766400 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:20:35.766376    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd\": container with ID starting with cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd not found: ID does not exist" containerID="cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd"
Sep 09 21:20:35.766457 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.766410    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd"} err="failed to get container status \"cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd\": rpc error: code = NotFound desc = could not find container \"cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd\": container with ID starting with cc1aa9e6fa8f0ba295f0c7ab4929fdfd956abef28886fe09bd8159605dfc08bd not found: ID does not exist"
Sep 09 21:20:35.768950 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:20:35.768927    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ac9fc3_0ac5_4a37_9e65_623a0b02bbad.slice/crio-26da57f08dd6ba4accd3916c9514498a91b68abb8f14ba6fc75cb46643635164\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ac9fc3_0ac5_4a37_9e65_623a0b02bbad.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 21:20:35.773607 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.773587    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"]
Sep 09 21:20:35.776145 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:35.776123    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-vmwb6"]
Sep 09 21:20:36.622552 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:36.622516    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:20:36.936798 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:36.936706    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" path="/var/lib/kubelet/pods/a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad/volumes"
Sep 09 21:20:39.581452 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581421    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"]
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581803    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kserve-container"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581816    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kserve-container"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581826    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kube-rbac-proxy"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581831    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kube-rbac-proxy"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581846    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="storage-initializer"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581852    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="storage-initializer"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581861    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kserve-container"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581868    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kserve-container"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581880    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="storage-initializer"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581887    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="storage-initializer"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581899    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kube-rbac-proxy"
Sep 09 21:20:39.581915 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581904    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kube-rbac-proxy"
Sep 09 21:20:39.582291 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581957    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kserve-container"
Sep 09 21:20:39.582291 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581966    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kserve-container"
Sep 09 21:20:39.582291 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581973    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="dad1f544-56aa-4039-be5a-117fc2223507" containerName="kube-rbac-proxy"
Sep 09 21:20:39.582291 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.581978    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ac9fc3-0ac5-4a37-9e65-623a0b02bbad" containerName="kube-rbac-proxy"
Sep 09 21:20:39.615327 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.615297    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"]
Sep 09 21:20:39.615474 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.615420    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.619148 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.619129    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 09 21:20:39.637571 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.637553    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 09 21:20:39.641334 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.641318    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.644561 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.644544    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 09 21:20:39.652456 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.652433    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"]
Sep 09 21:20:39.677350 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.677327    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"]
Sep 09 21:20:39.697297 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.697412 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697301    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.697412 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697335    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aa5472-9537-4897-a4c4-7b1debe5a3c5-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.697412 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697355    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49zxr\" (UniqueName: \"kubernetes.io/projected/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-api-access-49zxr\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.697412 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.697591 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697446    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.697591 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.697591 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mhs4\" (UniqueName: \"kubernetes.io/projected/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-api-access-4mhs4\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.697591 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.697591 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.697550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.798928 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.798888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aa5472-9537-4897-a4c4-7b1debe5a3c5-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.798928 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.798934    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49zxr\" (UniqueName: \"kubernetes.io/projected/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-api-access-49zxr\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.799188 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.798953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.799188 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.798980    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.799188 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.799188 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mhs4\" (UniqueName: \"kubernetes.io/projected/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-api-access-4mhs4\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.799188 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799113    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.799188 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799138    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.799491 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799294    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.799491 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799336    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.799491 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kserve-provision-location\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.799642 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.799917 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799894    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-cabundle-cert\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.800030 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.799970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.800030 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.800026    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.800316 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.800210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.801632 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.801614    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aa5472-9537-4897-a4c4-7b1debe5a3c5-proxy-tls\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.801977 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.801959    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-proxy-tls\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.808384 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.808352    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mhs4\" (UniqueName: \"kubernetes.io/projected/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-api-access-4mhs4\") pod \"isvc-canary-create-v2-predictor-5cd44d6886-7qktq\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.810050 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.810024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49zxr\" (UniqueName: \"kubernetes.io/projected/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-api-access-49zxr\") pod \"isvc-canary-create-predictor-854596548c-24qm7\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:39.925780 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.925723    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:39.951064 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:39.951033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:40.240202 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:40.240169    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"]
Sep 09 21:20:40.242123 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:20:40.242098    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52aa5472_9537_4897_a4c4_7b1debe5a3c5.slice/crio-c4d74260149363b8247c37310f4405d79cf74f2e1fb605db1a227f1edf7b7cd8 WatchSource:0}: Error finding container c4d74260149363b8247c37310f4405d79cf74f2e1fb605db1a227f1edf7b7cd8: Status 404 returned error can't find the container with id c4d74260149363b8247c37310f4405d79cf74f2e1fb605db1a227f1edf7b7cd8
Sep 09 21:20:40.268299 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:40.268269    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"]
Sep 09 21:20:40.269707 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:20:40.269673    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ed4980_09d0_4ea8_b7ff_50be6f4e231d.slice/crio-92657767c3a549636bf74e21b417f8fec4b370dd545828a8b2482adf5ba951a8 WatchSource:0}: Error finding container 92657767c3a549636bf74e21b417f8fec4b370dd545828a8b2482adf5ba951a8: Status 404 returned error can't find the container with id 92657767c3a549636bf74e21b417f8fec4b370dd545828a8b2482adf5ba951a8
Sep 09 21:20:40.762494 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:40.762443    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" event={"ID":"52aa5472-9537-4897-a4c4-7b1debe5a3c5","Type":"ContainerStarted","Data":"cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb"}
Sep 09 21:20:40.763013 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:40.762569    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" event={"ID":"52aa5472-9537-4897-a4c4-7b1debe5a3c5","Type":"ContainerStarted","Data":"c4d74260149363b8247c37310f4405d79cf74f2e1fb605db1a227f1edf7b7cd8"}
Sep 09 21:20:40.764271 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:40.764241    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" event={"ID":"32ed4980-09d0-4ea8-b7ff-50be6f4e231d","Type":"ContainerStarted","Data":"e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359"}
Sep 09 21:20:40.764411 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:40.764302    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" event={"ID":"32ed4980-09d0-4ea8-b7ff-50be6f4e231d","Type":"ContainerStarted","Data":"92657767c3a549636bf74e21b417f8fec4b370dd545828a8b2482adf5ba951a8"}
Sep 09 21:20:44.785797 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:44.785745    2580 generic.go:356] "Generic (PLEG): container finished" podID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerID="e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359" exitCode=0
Sep 09 21:20:44.786218 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:44.785821    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" event={"ID":"32ed4980-09d0-4ea8-b7ff-50be6f4e231d","Type":"ContainerDied","Data":"e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359"}
Sep 09 21:20:44.787483 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:44.787462    2580 generic.go:356] "Generic (PLEG): container finished" podID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerID="cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb" exitCode=0
Sep 09 21:20:44.787573 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:44.787527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" event={"ID":"52aa5472-9537-4897-a4c4-7b1debe5a3c5","Type":"ContainerDied","Data":"cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb"}
Sep 09 21:20:45.795611 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:45.795575    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" event={"ID":"32ed4980-09d0-4ea8-b7ff-50be6f4e231d","Type":"ContainerStarted","Data":"a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2"}
Sep 09 21:20:45.796093 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:45.795637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" event={"ID":"32ed4980-09d0-4ea8-b7ff-50be6f4e231d","Type":"ContainerStarted","Data":"2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227"}
Sep 09 21:20:45.798087 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:45.798066    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" event={"ID":"52aa5472-9537-4897-a4c4-7b1debe5a3c5","Type":"ContainerStarted","Data":"b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8"}
Sep 09 21:20:45.798181 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:45.798116    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" event={"ID":"52aa5472-9537-4897-a4c4-7b1debe5a3c5","Type":"ContainerStarted","Data":"76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42"}
Sep 09 21:20:45.798311 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:45.798293    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:45.798370 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:45.798337    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:45.800022 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:45.799999    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:20:45.822022 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:45.821977    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podStartSLOduration=2.782739358 podStartE2EDuration="6.821963582s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.039224224s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:45.819082383 +0000 UTC m=+1735.380109694" watchObservedRunningTime="2026-09-09 21:20:45.821963582 +0000 UTC m=+1735.382990899"
Sep 09 21:20:45.848105 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:45.848068    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podStartSLOduration=3.033944191 podStartE2EDuration="6.848054853s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.814110662s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:20:45.847701913 +0000 UTC m=+1735.408729230" watchObservedRunningTime="2026-09-09 21:20:45.848054853 +0000 UTC m=+1735.409082225"
Sep 09 21:20:46.622470 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:46.622428    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:20:46.803709 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:46.803666    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:20:50.796064 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:50.796018    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:50.796568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:50.796500    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:50.797505 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:50.797474    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:20:50.801039 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:50.801016    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:20:50.824216 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:50.824187    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:20:51.808498 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:51.808472    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:20:51.808991 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:51.808964    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:20:51.828575 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:51.828540    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:20:56.622289 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:20:56.622253    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:21:01.809311 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:01.809273    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:21:01.829113 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:01.829084    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:21:06.622190 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:06.622151    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:21:11.809798 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:11.809739    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:21:11.828969 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:11.828934    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:21:16.622476 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:16.622439    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:21:21.809643 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:21.809608    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:21:21.829171 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:21.829128    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:21:26.622946 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:26.622912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:21:31.808984 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:31.808947    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:21:31.828833 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:31.828790    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:21:32.978400 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:32.978361    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"]
Sep 09 21:21:32.978904 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:32.978584    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" containerID="cri-o://b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42" gracePeriod=30
Sep 09 21:21:32.978904 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:32.978622    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kube-rbac-proxy" containerID="cri-o://ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65" gracePeriod=30
Sep 09 21:21:34.045607 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:34.045573    2580 generic.go:356] "Generic (PLEG): container finished" podID="0dd50125-8daa-420c-8182-34efc7182614" containerID="ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65" exitCode=2
Sep 09 21:21:34.046015 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:34.045648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" event={"ID":"0dd50125-8daa-420c-8182-34efc7182614","Type":"ContainerDied","Data":"ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65"}
Sep 09 21:21:36.616190 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:36.616146    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 09 21:21:36.622110 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:36.622075    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 09 21:21:38.022907 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.022883    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:21:38.070811 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.070783    2580 generic.go:356] "Generic (PLEG): container finished" podID="0dd50125-8daa-420c-8182-34efc7182614" containerID="b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42" exitCode=0
Sep 09 21:21:38.070978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.070862    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"
Sep 09 21:21:38.070978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.070867    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" event={"ID":"0dd50125-8daa-420c-8182-34efc7182614","Type":"ContainerDied","Data":"b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42"}
Sep 09 21:21:38.070978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.070928    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8" event={"ID":"0dd50125-8daa-420c-8182-34efc7182614","Type":"ContainerDied","Data":"b8e0f35cc4669fc07226b99b5c0ccc882f83a54e92ed49071c8f76a034a6252e"}
Sep 09 21:21:38.070978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.070956    2580 scope.go:117] "RemoveContainer" containerID="ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65"
Sep 09 21:21:38.080196 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.080178    2580 scope.go:117] "RemoveContainer" containerID="b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42"
Sep 09 21:21:38.088599 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.088568    2580 scope.go:117] "RemoveContainer" containerID="63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741"
Sep 09 21:21:38.096441 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.096422    2580 scope.go:117] "RemoveContainer" containerID="ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65"
Sep 09 21:21:38.096717 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:21:38.096693    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65\": container with ID starting with ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65 not found: ID does not exist" containerID="ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65"
Sep 09 21:21:38.096789 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.096727    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65"} err="failed to get container status \"ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65\": rpc error: code = NotFound desc = could not find container \"ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65\": container with ID starting with ce8fe94171cc9cf018a849b1bfd70d1cfc3116edfe5175b731d2049fb11daa65 not found: ID does not exist"
Sep 09 21:21:38.096789 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.096744    2580 scope.go:117] "RemoveContainer" containerID="b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42"
Sep 09 21:21:38.097004 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:21:38.096984    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42\": container with ID starting with b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42 not found: ID does not exist" containerID="b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42"
Sep 09 21:21:38.097044 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.097012    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42"} err="failed to get container status \"b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42\": rpc error: code = NotFound desc = could not find container \"b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42\": container with ID starting with b2edb4c0131c85aa5e1532b65e59c26aefecd1f2eb32e2666e5ddee4a2298c42 not found: ID does not exist"
Sep 09 21:21:38.097044 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.097027    2580 scope.go:117] "RemoveContainer" containerID="63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741"
Sep 09 21:21:38.097307 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:21:38.097287    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741\": container with ID starting with 63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741 not found: ID does not exist" containerID="63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741"
Sep 09 21:21:38.097361 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.097312    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741"} err="failed to get container status \"63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741\": rpc error: code = NotFound desc = could not find container \"63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741\": container with ID starting with 63f1eb101eba8894bed3efc07ba8fac1fa1b50766dace9c2b7f2ef715f12e741 not found: ID does not exist"
Sep 09 21:21:38.120517 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.120440    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dd50125-8daa-420c-8182-34efc7182614-kserve-provision-location\") pod \"0dd50125-8daa-420c-8182-34efc7182614\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") "
Sep 09 21:21:38.120635 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.120537    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-cabundle-cert\") pod \"0dd50125-8daa-420c-8182-34efc7182614\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") "
Sep 09 21:21:38.120635 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.120565    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd50125-8daa-420c-8182-34efc7182614-proxy-tls\") pod \"0dd50125-8daa-420c-8182-34efc7182614\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") "
Sep 09 21:21:38.120635 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.120615    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dstql\" (UniqueName: \"kubernetes.io/projected/0dd50125-8daa-420c-8182-34efc7182614-kube-api-access-dstql\") pod \"0dd50125-8daa-420c-8182-34efc7182614\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") "
Sep 09 21:21:38.120800 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.120642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-kube-rbac-proxy-sar-config\") pod \"0dd50125-8daa-420c-8182-34efc7182614\" (UID: \"0dd50125-8daa-420c-8182-34efc7182614\") "
Sep 09 21:21:38.120976 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.120944    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0dd50125-8daa-420c-8182-34efc7182614" (UID: "0dd50125-8daa-420c-8182-34efc7182614"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:21:38.121204 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.121162    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dd50125-8daa-420c-8182-34efc7182614-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0dd50125-8daa-420c-8182-34efc7182614" (UID: "0dd50125-8daa-420c-8182-34efc7182614"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:21:38.121565 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.121535    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0dd50125-8daa-420c-8182-34efc7182614" (UID: "0dd50125-8daa-420c-8182-34efc7182614"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:21:38.123076 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.123049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd50125-8daa-420c-8182-34efc7182614-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0dd50125-8daa-420c-8182-34efc7182614" (UID: "0dd50125-8daa-420c-8182-34efc7182614"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:21:38.123210 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.123190    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd50125-8daa-420c-8182-34efc7182614-kube-api-access-dstql" (OuterVolumeSpecName: "kube-api-access-dstql") pod "0dd50125-8daa-420c-8182-34efc7182614" (UID: "0dd50125-8daa-420c-8182-34efc7182614"). InnerVolumeSpecName "kube-api-access-dstql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:21:38.221687 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.221654    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:21:38.221687 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.221681    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0dd50125-8daa-420c-8182-34efc7182614-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:21:38.221687 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.221690    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dstql\" (UniqueName: \"kubernetes.io/projected/0dd50125-8daa-420c-8182-34efc7182614-kube-api-access-dstql\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:21:38.221956 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.221700    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0dd50125-8daa-420c-8182-34efc7182614-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:21:38.221956 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.221710    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0dd50125-8daa-420c-8182-34efc7182614-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:21:38.400668 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.400637    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"]
Sep 09 21:21:38.421065 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.421037    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-64bfb47598-k2nn8"]
Sep 09 21:21:38.936410 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:38.936381    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dd50125-8daa-420c-8182-34efc7182614" path="/var/lib/kubelet/pods/0dd50125-8daa-420c-8182-34efc7182614/volumes"
Sep 09 21:21:41.809151 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:41.809063    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:21:41.828968 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:41.828936    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:21:51.809192 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:51.809155    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:21:51.829301 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:51.829258    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:21:52.930978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:21:52.930933    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:22:01.829305 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:01.829262    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:22:02.936232 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:02.936197    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:22:05.037596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:05.037476    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:22:05.037596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:05.037480    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:22:09.660083 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:09.660048    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"]
Sep 09 21:22:09.660507 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:09.660321    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" containerID="cri-o://2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227" gracePeriod=30
Sep 09 21:22:09.660507 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:09.660351    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kube-rbac-proxy" containerID="cri-o://a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2" gracePeriod=30
Sep 09 21:22:09.667492 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:09.667468    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"]
Sep 09 21:22:09.667725 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:09.667690    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" containerID="cri-o://76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42" gracePeriod=30
Sep 09 21:22:09.667911 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:09.667857    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kube-rbac-proxy" containerID="cri-o://b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8" gracePeriod=30
Sep 09 21:22:10.235200 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:10.235168    2580 generic.go:356] "Generic (PLEG): container finished" podID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerID="a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2" exitCode=2
Sep 09 21:22:10.235383 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:10.235245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" event={"ID":"32ed4980-09d0-4ea8-b7ff-50be6f4e231d","Type":"ContainerDied","Data":"a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2"}
Sep 09 21:22:10.237307 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:10.237289    2580 generic.go:356] "Generic (PLEG): container finished" podID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerID="b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8" exitCode=2
Sep 09 21:22:10.237390 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:10.237368    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" event={"ID":"52aa5472-9537-4897-a4c4-7b1debe5a3c5","Type":"ContainerDied","Data":"b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8"}
Sep 09 21:22:10.801538 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:10.801493    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 09 21:22:11.804908 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:11.804862    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 09 21:22:11.828534 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:11.828498    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 09 21:22:12.931187 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:12.931138    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 09 21:22:14.619751 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.619706    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:22:14.622579 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.622561    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:22:14.734115 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734030    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49zxr\" (UniqueName: \"kubernetes.io/projected/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-api-access-49zxr\") pod \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") "
Sep 09 21:22:14.734115 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734093    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-cabundle-cert\") pod \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") "
Sep 09 21:22:14.734339 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734130    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aa5472-9537-4897-a4c4-7b1debe5a3c5-proxy-tls\") pod \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") "
Sep 09 21:22:14.734339 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734147    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-cabundle-cert\") pod \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") "
Sep 09 21:22:14.734339 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734182    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-rbac-proxy-sar-config\") pod \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") "
Sep 09 21:22:14.734339 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734219    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kserve-provision-location\") pod \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") "
Sep 09 21:22:14.734339 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kserve-provision-location\") pod \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") "
Sep 09 21:22:14.734339 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734295    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-rbac-proxy-sar-config\") pod \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") "
Sep 09 21:22:14.734339 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734340    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-proxy-tls\") pod \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\" (UID: \"32ed4980-09d0-4ea8-b7ff-50be6f4e231d\") "
Sep 09 21:22:14.734712 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734379    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mhs4\" (UniqueName: \"kubernetes.io/projected/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-api-access-4mhs4\") pod \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\" (UID: \"52aa5472-9537-4897-a4c4-7b1debe5a3c5\") "
Sep 09 21:22:14.734712 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734513    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "52aa5472-9537-4897-a4c4-7b1debe5a3c5" (UID: "52aa5472-9537-4897-a4c4-7b1debe5a3c5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:14.734712 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734513    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32ed4980-09d0-4ea8-b7ff-50be6f4e231d" (UID: "32ed4980-09d0-4ea8-b7ff-50be6f4e231d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:14.734712 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734548    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32ed4980-09d0-4ea8-b7ff-50be6f4e231d" (UID: "32ed4980-09d0-4ea8-b7ff-50be6f4e231d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:14.734712 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734640    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.734712 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734658    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.734712 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734671    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.735105 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734841    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32ed4980-09d0-4ea8-b7ff-50be6f4e231d" (UID: "32ed4980-09d0-4ea8-b7ff-50be6f4e231d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:14.735105 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.734880    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52aa5472-9537-4897-a4c4-7b1debe5a3c5" (UID: "52aa5472-9537-4897-a4c4-7b1debe5a3c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:14.735384 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.735174    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "52aa5472-9537-4897-a4c4-7b1debe5a3c5" (UID: "52aa5472-9537-4897-a4c4-7b1debe5a3c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:14.736725 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.736694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32ed4980-09d0-4ea8-b7ff-50be6f4e231d" (UID: "32ed4980-09d0-4ea8-b7ff-50be6f4e231d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:14.736884 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.736827    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-api-access-4mhs4" (OuterVolumeSpecName: "kube-api-access-4mhs4") pod "52aa5472-9537-4897-a4c4-7b1debe5a3c5" (UID: "52aa5472-9537-4897-a4c4-7b1debe5a3c5"). InnerVolumeSpecName "kube-api-access-4mhs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:14.737000 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.736980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52aa5472-9537-4897-a4c4-7b1debe5a3c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "52aa5472-9537-4897-a4c4-7b1debe5a3c5" (UID: "52aa5472-9537-4897-a4c4-7b1debe5a3c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:14.737064 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.737041    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-api-access-49zxr" (OuterVolumeSpecName: "kube-api-access-49zxr") pod "32ed4980-09d0-4ea8-b7ff-50be6f4e231d" (UID: "32ed4980-09d0-4ea8-b7ff-50be6f4e231d"). InnerVolumeSpecName "kube-api-access-49zxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:14.775197 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775166    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"]
Sep 09 21:22:14.775512 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775500    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kube-rbac-proxy"
Sep 09 21:22:14.775568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775514    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kube-rbac-proxy"
Sep 09 21:22:14.775568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775539    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container"
Sep 09 21:22:14.775568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775545    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container"
Sep 09 21:22:14.775568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775559    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="storage-initializer"
Sep 09 21:22:14.775568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775565    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="storage-initializer"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775571    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775576    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775583    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="storage-initializer"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775588    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="storage-initializer"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775595    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775600    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775607    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="storage-initializer"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775612    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="storage-initializer"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775617    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kube-rbac-proxy"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775622    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kube-rbac-proxy"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775630    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kube-rbac-proxy"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775635    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kube-rbac-proxy"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775687    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kserve-container"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775696    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kserve-container"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775702    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerName="kube-rbac-proxy"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775707    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerName="kube-rbac-proxy"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775712    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kserve-container"
Sep 09 21:22:14.775740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.775718    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd50125-8daa-420c-8182-34efc7182614" containerName="kube-rbac-proxy"
Sep 09 21:22:14.778551 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.778534    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.781630 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.781612    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 09 21:22:14.781731 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.781665    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 09 21:22:14.792495 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.792472    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"]
Sep 09 21:22:14.817597 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.817565    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"]
Sep 09 21:22:14.820572 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.820557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:14.829673 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.829642    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 09 21:22:14.835453 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835422    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.835581 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835466    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.835581 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835515    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4a93ef-3294-4e64-a961-16afefe14c52-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.835702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca4a93ef-3294-4e64-a961-16afefe14c52-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.835702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835628    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84q5l\" (UniqueName: \"kubernetes.io/projected/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-api-access-84q5l\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.835702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835690    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.835702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835701    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.835958 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835710    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.835958 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835723    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.835958 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835733    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mhs4\" (UniqueName: \"kubernetes.io/projected/52aa5472-9537-4897-a4c4-7b1debe5a3c5-kube-api-access-4mhs4\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.835958 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835740    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49zxr\" (UniqueName: \"kubernetes.io/projected/32ed4980-09d0-4ea8-b7ff-50be6f4e231d-kube-api-access-49zxr\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.835958 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.835748    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/52aa5472-9537-4897-a4c4-7b1debe5a3c5-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:22:14.836378 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.836358    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"]
Sep 09 21:22:14.936150 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.936150 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936146    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ca6757-691d-4cf2-a436-93de6f149ea0-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:14.936398 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.936398 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrn5\" (UniqueName: \"kubernetes.io/projected/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-api-access-znrn5\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:14.936398 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936307    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:14.936398 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936358    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4a93ef-3294-4e64-a961-16afefe14c52-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.936606 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936421    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca4a93ef-3294-4e64-a961-16afefe14c52-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.936606 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936449    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:14.936606 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:22:14.936459    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 09 21:22:14.936606 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:22:14.936520    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca4a93ef-3294-4e64-a961-16afefe14c52-proxy-tls podName:ca4a93ef-3294-4e64-a961-16afefe14c52 nodeName:}" failed. No retries permitted until 2026-09-09 21:22:15.436500762 +0000 UTC m=+1824.997528056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ca4a93ef-3294-4e64-a961-16afefe14c52-proxy-tls") pod "isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" (UID: "ca4a93ef-3294-4e64-a961-16afefe14c52") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found
Sep 09 21:22:14.936606 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84q5l\" (UniqueName: \"kubernetes.io/projected/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-api-access-84q5l\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.936606 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936574    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ca6757-691d-4cf2-a436-93de6f149ea0-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:14.937003 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936840    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca4a93ef-3294-4e64-a961-16afefe14c52-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.937003 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.937003 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.936957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:14.952409 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:14.952387    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84q5l\" (UniqueName: \"kubernetes.io/projected/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-api-access-84q5l\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:15.037432 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.037333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ca6757-691d-4cf2-a436-93de6f149ea0-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.037432 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.037384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrn5\" (UniqueName: \"kubernetes.io/projected/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-api-access-znrn5\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.037432 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.037416    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.037696 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.037454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.037696 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.037518    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ca6757-691d-4cf2-a436-93de6f149ea0-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.038002 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.037965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ca6757-691d-4cf2-a436-93de6f149ea0-kserve-provision-location\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.038219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.038197    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.038288 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.038200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-cabundle-cert\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.040047 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.040022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ca6757-691d-4cf2-a436-93de6f149ea0-proxy-tls\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.046701 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.046679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrn5\" (UniqueName: \"kubernetes.io/projected/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-api-access-znrn5\") pod \"isvc-canary-promote-predictor-65ff4fc5f5-ckt54\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.140452 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.140394    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:15.265021 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.264988    2580 generic.go:356] "Generic (PLEG): container finished" podID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" containerID="2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227" exitCode=0
Sep 09 21:22:15.265221 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.265061    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"
Sep 09 21:22:15.265221 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.265066    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" event={"ID":"32ed4980-09d0-4ea8-b7ff-50be6f4e231d","Type":"ContainerDied","Data":"2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227"}
Sep 09 21:22:15.265221 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.265142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7" event={"ID":"32ed4980-09d0-4ea8-b7ff-50be6f4e231d","Type":"ContainerDied","Data":"92657767c3a549636bf74e21b417f8fec4b370dd545828a8b2482adf5ba951a8"}
Sep 09 21:22:15.265221 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.265178    2580 scope.go:117] "RemoveContainer" containerID="a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2"
Sep 09 21:22:15.267617 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.267475    2580 generic.go:356] "Generic (PLEG): container finished" podID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" containerID="76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42" exitCode=0
Sep 09 21:22:15.267617 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.267529    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" event={"ID":"52aa5472-9537-4897-a4c4-7b1debe5a3c5","Type":"ContainerDied","Data":"76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42"}
Sep 09 21:22:15.267617 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.267580    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq" event={"ID":"52aa5472-9537-4897-a4c4-7b1debe5a3c5","Type":"ContainerDied","Data":"c4d74260149363b8247c37310f4405d79cf74f2e1fb605db1a227f1edf7b7cd8"}
Sep 09 21:22:15.267880 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.267637    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"
Sep 09 21:22:15.274319 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.274304    2580 scope.go:117] "RemoveContainer" containerID="2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227"
Sep 09 21:22:15.282376 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.282358    2580 scope.go:117] "RemoveContainer" containerID="e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359"
Sep 09 21:22:15.288617 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.288555    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"]
Sep 09 21:22:15.290625 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.290604    2580 scope.go:117] "RemoveContainer" containerID="a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2"
Sep 09 21:22:15.291048 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:22:15.291012    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2\": container with ID starting with a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2 not found: ID does not exist" containerID="a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2"
Sep 09 21:22:15.291167 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.291050    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2"} err="failed to get container status \"a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2\": rpc error: code = NotFound desc = could not find container \"a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2\": container with ID starting with a5d7e0e1a89be8ad0e82875e66e722a78926981abfe57ebfc2b1e819cb0921e2 not found: ID does not exist"
Sep 09 21:22:15.291167 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.291076    2580 scope.go:117] "RemoveContainer" containerID="2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227"
Sep 09 21:22:15.291567 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:22:15.291543    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227\": container with ID starting with 2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227 not found: ID does not exist" containerID="2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227"
Sep 09 21:22:15.291659 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.291575    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227"} err="failed to get container status \"2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227\": rpc error: code = NotFound desc = could not find container \"2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227\": container with ID starting with 2af0945af793d3d5790045c7c3932e86af3a18671cf1b6893c93dcbb4c1ba227 not found: ID does not exist"
Sep 09 21:22:15.291659 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.291592    2580 scope.go:117] "RemoveContainer" containerID="e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359"
Sep 09 21:22:15.291933 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:22:15.291917    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359\": container with ID starting with e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359 not found: ID does not exist" containerID="e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359"
Sep 09 21:22:15.291980 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.291939    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359"} err="failed to get container status \"e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359\": rpc error: code = NotFound desc = could not find container \"e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359\": container with ID starting with e477e6ce3357c8dc941bc3c7d42c46fc95a7d617d62e55386c6875c3a5334359 not found: ID does not exist"
Sep 09 21:22:15.291980 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.291967    2580 scope.go:117] "RemoveContainer" containerID="b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8"
Sep 09 21:22:15.293501 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.293484    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-854596548c-24qm7"]
Sep 09 21:22:15.299693 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.299675    2580 scope.go:117] "RemoveContainer" containerID="76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42"
Sep 09 21:22:15.305442 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.305418    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"]
Sep 09 21:22:15.308016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.307998    2580 scope.go:117] "RemoveContainer" containerID="cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb"
Sep 09 21:22:15.308366 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.308349    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-5cd44d6886-7qktq"]
Sep 09 21:22:15.315551 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.315534    2580 scope.go:117] "RemoveContainer" containerID="b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8"
Sep 09 21:22:15.315942 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:22:15.315923    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8\": container with ID starting with b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8 not found: ID does not exist" containerID="b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8"
Sep 09 21:22:15.316035 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.315972    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8"} err="failed to get container status \"b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8\": rpc error: code = NotFound desc = could not find container \"b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8\": container with ID starting with b08905a5b812f322fe07127ede2d8f9bc1f28149160e12cf5f2bc6f04cd30ce8 not found: ID does not exist"
Sep 09 21:22:15.316035 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.315997    2580 scope.go:117] "RemoveContainer" containerID="76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42"
Sep 09 21:22:15.316273 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:22:15.316254    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42\": container with ID starting with 76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42 not found: ID does not exist" containerID="76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42"
Sep 09 21:22:15.316316 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.316280    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42"} err="failed to get container status \"76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42\": rpc error: code = NotFound desc = could not find container \"76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42\": container with ID starting with 76a688e70c7c332abdb61c06c2ab9f886cbff5aaf9fed5220ccf23d82d32ee42 not found: ID does not exist"
Sep 09 21:22:15.316316 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.316296    2580 scope.go:117] "RemoveContainer" containerID="cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb"
Sep 09 21:22:15.316544 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:22:15.316525    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb\": container with ID starting with cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb not found: ID does not exist" containerID="cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb"
Sep 09 21:22:15.316588 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.316550    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb"} err="failed to get container status \"cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb\": rpc error: code = NotFound desc = could not find container \"cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb\": container with ID starting with cd5c210bb0b62c27c0f0c2e391d3c66f1307bf9e153fea57c0ab8bfa00c9dacb not found: ID does not exist"
Sep 09 21:22:15.422206 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.422180    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"]
Sep 09 21:22:15.423942 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:22:15.423919    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ca6757_691d_4cf2_a436_93de6f149ea0.slice/crio-ca4c7b73f67d7d881e7d4b35844642b4eb15f7492a127cdee3bfdc35c149b8e7 WatchSource:0}: Error finding container ca4c7b73f67d7d881e7d4b35844642b4eb15f7492a127cdee3bfdc35c149b8e7: Status 404 returned error can't find the container with id ca4c7b73f67d7d881e7d4b35844642b4eb15f7492a127cdee3bfdc35c149b8e7
Sep 09 21:22:15.442712 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.442680    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4a93ef-3294-4e64-a961-16afefe14c52-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:15.445166 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.445148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4a93ef-3294-4e64-a961-16afefe14c52-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b8964cf76-q7dwh\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:15.689406 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.689364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:15.972350 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:15.972322    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"]
Sep 09 21:22:15.975337 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:22:15.975314    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4a93ef_3294_4e64_a961_16afefe14c52.slice/crio-cc7a692fc8b9963d5a31d91cf148f6e44ad5da74af216d739335380dc93fc0bd WatchSource:0}: Error finding container cc7a692fc8b9963d5a31d91cf148f6e44ad5da74af216d739335380dc93fc0bd: Status 404 returned error can't find the container with id cc7a692fc8b9963d5a31d91cf148f6e44ad5da74af216d739335380dc93fc0bd
Sep 09 21:22:16.275375 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:16.275271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" event={"ID":"ca4a93ef-3294-4e64-a961-16afefe14c52","Type":"ContainerStarted","Data":"22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f"}
Sep 09 21:22:16.275375 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:16.275348    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" event={"ID":"ca4a93ef-3294-4e64-a961-16afefe14c52","Type":"ContainerStarted","Data":"cc7a692fc8b9963d5a31d91cf148f6e44ad5da74af216d739335380dc93fc0bd"}
Sep 09 21:22:16.277976 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:16.277950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" event={"ID":"f0ca6757-691d-4cf2-a436-93de6f149ea0","Type":"ContainerStarted","Data":"b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279"}
Sep 09 21:22:16.278097 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:16.278002    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" event={"ID":"f0ca6757-691d-4cf2-a436-93de6f149ea0","Type":"ContainerStarted","Data":"ca4c7b73f67d7d881e7d4b35844642b4eb15f7492a127cdee3bfdc35c149b8e7"}
Sep 09 21:22:16.939411 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:16.939376    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32ed4980-09d0-4ea8-b7ff-50be6f4e231d" path="/var/lib/kubelet/pods/32ed4980-09d0-4ea8-b7ff-50be6f4e231d/volumes"
Sep 09 21:22:16.940102 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:16.940081    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52aa5472-9537-4897-a4c4-7b1debe5a3c5" path="/var/lib/kubelet/pods/52aa5472-9537-4897-a4c4-7b1debe5a3c5/volumes"
Sep 09 21:22:20.300523 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:20.300490    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerID="22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f" exitCode=0
Sep 09 21:22:20.300968 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:20.300564    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" event={"ID":"ca4a93ef-3294-4e64-a961-16afefe14c52","Type":"ContainerDied","Data":"22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f"}
Sep 09 21:22:20.302210 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:20.302188    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerID="b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279" exitCode=0
Sep 09 21:22:20.302308 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:20.302253    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" event={"ID":"f0ca6757-691d-4cf2-a436-93de6f149ea0","Type":"ContainerDied","Data":"b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279"}
Sep 09 21:22:21.309200 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.309155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" event={"ID":"ca4a93ef-3294-4e64-a961-16afefe14c52","Type":"ContainerStarted","Data":"64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0"}
Sep 09 21:22:21.309641 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.309243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" event={"ID":"ca4a93ef-3294-4e64-a961-16afefe14c52","Type":"ContainerStarted","Data":"ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57"}
Sep 09 21:22:21.309641 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.309441    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:21.311485 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.311462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" event={"ID":"f0ca6757-691d-4cf2-a436-93de6f149ea0","Type":"ContainerStarted","Data":"45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0"}
Sep 09 21:22:21.311599 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.311510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" event={"ID":"f0ca6757-691d-4cf2-a436-93de6f149ea0","Type":"ContainerStarted","Data":"a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93"}
Sep 09 21:22:21.311812 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.311795    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:21.311859 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.311836    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:21.313470 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.313445    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:22:21.331686 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.331617    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podStartSLOduration=3.5596672959999998 podStartE2EDuration="7.331599704s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771932408s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:21.330878113 +0000 UTC m=+1830.891905444" watchObservedRunningTime="2026-09-09 21:22:21.331599704 +0000 UTC m=+1830.892627022"
Sep 09 21:22:21.353125 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:21.353072    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podStartSLOduration=3.37529421 podStartE2EDuration="7.35305246s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.97775825s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:21.350827635 +0000 UTC m=+1830.911854951" watchObservedRunningTime="2026-09-09 21:22:21.35305246 +0000 UTC m=+1830.914079778"
Sep 09 21:22:22.317628 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:22.317530    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:22:22.317628 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:22.317610    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:22.318790 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:22.318752    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:22:23.321836 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:23.321795    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:22:27.321531 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:27.321504    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:22:27.322039 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:27.322011    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:22:28.332363 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:28.332332    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:22:28.332881 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:28.332845    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:22:37.322795 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:37.322731    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:22:38.333529 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:38.333484    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:22:47.322342 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:47.322299    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:22:48.333134 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:48.333097    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:22:57.322396 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:57.322356    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:22:58.333690 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:22:58.333651    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:23:07.322514 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:07.322472    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:23:08.333052 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:08.333008    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:23:17.322720 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:17.322630    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:23:18.333327 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:18.333286    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:23:27.322564 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:27.322524    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:23:28.333637 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:28.333595    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:23:33.930478 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:33.930439    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 09 21:23:34.935707 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:34.935681    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:23:43.931649 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:43.931614    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:23:44.740572 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:44.740537    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"]
Sep 09 21:23:44.740872 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:44.740843    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" containerID="cri-o://a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93" gracePeriod=30
Sep 09 21:23:44.740975 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:44.740852    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kube-rbac-proxy" containerID="cri-o://45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0" gracePeriod=30
Sep 09 21:23:44.930191 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:44.930152    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 09 21:23:45.725801 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:45.725740    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerID="45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0" exitCode=2
Sep 09 21:23:45.726199 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:45.725811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" event={"ID":"f0ca6757-691d-4cf2-a436-93de6f149ea0","Type":"ContainerDied","Data":"45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0"}
Sep 09 21:23:47.318250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:47.318212    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Sep 09 21:23:49.488679 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.488651    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:23:49.570534 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.570453    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ca6757-691d-4cf2-a436-93de6f149ea0-kserve-provision-location\") pod \"f0ca6757-691d-4cf2-a436-93de6f149ea0\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") "
Sep 09 21:23:49.570534 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.570503    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-rbac-proxy-sar-config\") pod \"f0ca6757-691d-4cf2-a436-93de6f149ea0\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") "
Sep 09 21:23:49.570534 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.570528    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ca6757-691d-4cf2-a436-93de6f149ea0-proxy-tls\") pod \"f0ca6757-691d-4cf2-a436-93de6f149ea0\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") "
Sep 09 21:23:49.570854 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.570589    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znrn5\" (UniqueName: \"kubernetes.io/projected/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-api-access-znrn5\") pod \"f0ca6757-691d-4cf2-a436-93de6f149ea0\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") "
Sep 09 21:23:49.570854 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.570604    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-cabundle-cert\") pod \"f0ca6757-691d-4cf2-a436-93de6f149ea0\" (UID: \"f0ca6757-691d-4cf2-a436-93de6f149ea0\") "
Sep 09 21:23:49.570979 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.570850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ca6757-691d-4cf2-a436-93de6f149ea0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0ca6757-691d-4cf2-a436-93de6f149ea0" (UID: "f0ca6757-691d-4cf2-a436-93de6f149ea0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:23:49.571019 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.570970    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0ca6757-691d-4cf2-a436-93de6f149ea0" (UID: "f0ca6757-691d-4cf2-a436-93de6f149ea0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:23:49.571057 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.571020    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f0ca6757-691d-4cf2-a436-93de6f149ea0" (UID: "f0ca6757-691d-4cf2-a436-93de6f149ea0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:23:49.572791 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.572771    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-api-access-znrn5" (OuterVolumeSpecName: "kube-api-access-znrn5") pod "f0ca6757-691d-4cf2-a436-93de6f149ea0" (UID: "f0ca6757-691d-4cf2-a436-93de6f149ea0"). InnerVolumeSpecName "kube-api-access-znrn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:23:49.572791 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.572755    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ca6757-691d-4cf2-a436-93de6f149ea0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0ca6757-691d-4cf2-a436-93de6f149ea0" (UID: "f0ca6757-691d-4cf2-a436-93de6f149ea0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:23:49.671511 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.671474    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znrn5\" (UniqueName: \"kubernetes.io/projected/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-api-access-znrn5\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:23:49.671511 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.671504    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:23:49.671511 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.671514    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ca6757-691d-4cf2-a436-93de6f149ea0-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:23:49.671736 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.671523    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ca6757-691d-4cf2-a436-93de6f149ea0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:23:49.671736 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.671532    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ca6757-691d-4cf2-a436-93de6f149ea0-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:23:49.748387 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.748350    2580 generic.go:356] "Generic (PLEG): container finished" podID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerID="a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93" exitCode=0
Sep 09 21:23:49.748568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.748417    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"
Sep 09 21:23:49.748568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.748439    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" event={"ID":"f0ca6757-691d-4cf2-a436-93de6f149ea0","Type":"ContainerDied","Data":"a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93"}
Sep 09 21:23:49.748568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.748496    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54" event={"ID":"f0ca6757-691d-4cf2-a436-93de6f149ea0","Type":"ContainerDied","Data":"ca4c7b73f67d7d881e7d4b35844642b4eb15f7492a127cdee3bfdc35c149b8e7"}
Sep 09 21:23:49.748568 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.748526    2580 scope.go:117] "RemoveContainer" containerID="45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0"
Sep 09 21:23:49.757985 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.757965    2580 scope.go:117] "RemoveContainer" containerID="a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93"
Sep 09 21:23:49.765695 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.765678    2580 scope.go:117] "RemoveContainer" containerID="b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279"
Sep 09 21:23:49.773745 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.773729    2580 scope.go:117] "RemoveContainer" containerID="45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0"
Sep 09 21:23:49.774044 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:23:49.774024    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0\": container with ID starting with 45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0 not found: ID does not exist" containerID="45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0"
Sep 09 21:23:49.774102 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.774052    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0"} err="failed to get container status \"45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0\": rpc error: code = NotFound desc = could not find container \"45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0\": container with ID starting with 45af6325d99a4ed04800cc102494a8822f90b82c0d5f11143197c66eaca79eb0 not found: ID does not exist"
Sep 09 21:23:49.774102 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.774068    2580 scope.go:117] "RemoveContainer" containerID="a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93"
Sep 09 21:23:49.774292 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:23:49.774275    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93\": container with ID starting with a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93 not found: ID does not exist" containerID="a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93"
Sep 09 21:23:49.774352 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.774301    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93"} err="failed to get container status \"a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93\": rpc error: code = NotFound desc = could not find container \"a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93\": container with ID starting with a02692c2c17efd5bfff147b6abdc03a296ca16408f76493afc6e9a339f122e93 not found: ID does not exist"
Sep 09 21:23:49.774352 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.774323    2580 scope.go:117] "RemoveContainer" containerID="b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279"
Sep 09 21:23:49.774605 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:23:49.774589    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279\": container with ID starting with b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279 not found: ID does not exist" containerID="b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279"
Sep 09 21:23:49.774666 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.774608    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279"} err="failed to get container status \"b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279\": rpc error: code = NotFound desc = could not find container \"b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279\": container with ID starting with b68274f5a33b75f402481cfe4fbb8cbeadce4467a09554f835dc2a55d75fa279 not found: ID does not exist"
Sep 09 21:23:49.774827 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.774808    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"]
Sep 09 21:23:49.779939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:49.779919    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-65ff4fc5f5-ckt54"]
Sep 09 21:23:50.937161 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:23:50.937133    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" path="/var/lib/kubelet/pods/f0ca6757-691d-4cf2-a436-93de6f149ea0/volumes"
Sep 09 21:25:25.015068 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:25.015037    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"]
Sep 09 21:25:25.015517 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:25.015274    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container" containerID="cri-o://ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57" gracePeriod=30
Sep 09 21:25:25.015517 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:25.015306    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kube-rbac-proxy" containerID="cri-o://64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0" gracePeriod=30
Sep 09 21:25:25.195659 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:25.195619    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerID="64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0" exitCode=2
Sep 09 21:25:25.195869 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:25.195685    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" event={"ID":"ca4a93ef-3294-4e64-a961-16afefe14c52","Type":"ContainerDied","Data":"64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0"}
Sep 09 21:25:28.322566 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:28.322523    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": dial tcp 10.132.0.54:8643: connect: connection refused"
Sep 09 21:25:29.866259 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:29.866229    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:25:30.013135 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.013044    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84q5l\" (UniqueName: \"kubernetes.io/projected/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-api-access-84q5l\") pod \"ca4a93ef-3294-4e64-a961-16afefe14c52\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") "
Sep 09 21:25:30.013135 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.013096    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-rbac-proxy-sar-config\") pod \"ca4a93ef-3294-4e64-a961-16afefe14c52\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") "
Sep 09 21:25:30.013135 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.013135    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca4a93ef-3294-4e64-a961-16afefe14c52-kserve-provision-location\") pod \"ca4a93ef-3294-4e64-a961-16afefe14c52\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") "
Sep 09 21:25:30.013425 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.013197    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-cabundle-cert\") pod \"ca4a93ef-3294-4e64-a961-16afefe14c52\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") "
Sep 09 21:25:30.013425 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.013225    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4a93ef-3294-4e64-a961-16afefe14c52-proxy-tls\") pod \"ca4a93ef-3294-4e64-a961-16afefe14c52\" (UID: \"ca4a93ef-3294-4e64-a961-16afefe14c52\") "
Sep 09 21:25:30.013515 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.013442    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4a93ef-3294-4e64-a961-16afefe14c52-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ca4a93ef-3294-4e64-a961-16afefe14c52" (UID: "ca4a93ef-3294-4e64-a961-16afefe14c52"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:25:30.013515 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.013498    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ca4a93ef-3294-4e64-a961-16afefe14c52" (UID: "ca4a93ef-3294-4e64-a961-16afefe14c52"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:25:30.013605 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.013579    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ca4a93ef-3294-4e64-a961-16afefe14c52" (UID: "ca4a93ef-3294-4e64-a961-16afefe14c52"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:25:30.015418 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.015392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-api-access-84q5l" (OuterVolumeSpecName: "kube-api-access-84q5l") pod "ca4a93ef-3294-4e64-a961-16afefe14c52" (UID: "ca4a93ef-3294-4e64-a961-16afefe14c52"). InnerVolumeSpecName "kube-api-access-84q5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:25:30.015418 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.015392    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca4a93ef-3294-4e64-a961-16afefe14c52-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ca4a93ef-3294-4e64-a961-16afefe14c52" (UID: "ca4a93ef-3294-4e64-a961-16afefe14c52"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:25:30.114910 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.114875    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ca4a93ef-3294-4e64-a961-16afefe14c52-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:25:30.114910 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.114903    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:25:30.114910 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.114913    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ca4a93ef-3294-4e64-a961-16afefe14c52-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:25:30.115133 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.114922    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84q5l\" (UniqueName: \"kubernetes.io/projected/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-api-access-84q5l\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:25:30.115133 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.114932    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ca4a93ef-3294-4e64-a961-16afefe14c52-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:25:30.220556 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.220517    2580 generic.go:356] "Generic (PLEG): container finished" podID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerID="ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57" exitCode=0
Sep 09 21:25:30.220702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.220584    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"
Sep 09 21:25:30.220702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.220597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" event={"ID":"ca4a93ef-3294-4e64-a961-16afefe14c52","Type":"ContainerDied","Data":"ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57"}
Sep 09 21:25:30.220702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.220661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh" event={"ID":"ca4a93ef-3294-4e64-a961-16afefe14c52","Type":"ContainerDied","Data":"cc7a692fc8b9963d5a31d91cf148f6e44ad5da74af216d739335380dc93fc0bd"}
Sep 09 21:25:30.220702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.220690    2580 scope.go:117] "RemoveContainer" containerID="64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0"
Sep 09 21:25:30.230100 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.230082    2580 scope.go:117] "RemoveContainer" containerID="ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57"
Sep 09 21:25:30.238324 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.238306    2580 scope.go:117] "RemoveContainer" containerID="22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f"
Sep 09 21:25:30.245889 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.245730    2580 scope.go:117] "RemoveContainer" containerID="64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0"
Sep 09 21:25:30.246202 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:25:30.246151    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0\": container with ID starting with 64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0 not found: ID does not exist" containerID="64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0"
Sep 09 21:25:30.246202 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.246185    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0"} err="failed to get container status \"64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0\": rpc error: code = NotFound desc = could not find container \"64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0\": container with ID starting with 64ebd9638a5ca2085a56eedbaf7788b02c47d7a81f040f24bd0b3020a2346ea0 not found: ID does not exist"
Sep 09 21:25:30.246672 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.246208    2580 scope.go:117] "RemoveContainer" containerID="ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57"
Sep 09 21:25:30.246745 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:25:30.246694    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57\": container with ID starting with ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57 not found: ID does not exist" containerID="ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57"
Sep 09 21:25:30.246856 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.246748    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57"} err="failed to get container status \"ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57\": rpc error: code = NotFound desc = could not find container \"ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57\": container with ID starting with ad61691f61af0fe0dc7ea4bdb98069b13ca36a9648fcade7eda62c803eb77e57 not found: ID does not exist"
Sep 09 21:25:30.246856 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.246794    2580 scope.go:117] "RemoveContainer" containerID="22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f"
Sep 09 21:25:30.247168 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:25:30.247128    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f\": container with ID starting with 22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f not found: ID does not exist" containerID="22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f"
Sep 09 21:25:30.247168 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.247156    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f"} err="failed to get container status \"22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f\": rpc error: code = NotFound desc = could not find container \"22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f\": container with ID starting with 22d44273bcffa10a85a74d5c4365ad5b0825a3aa89197d58175a372c3467459f not found: ID does not exist"
Sep 09 21:25:30.248749 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.248727    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"]
Sep 09 21:25:30.252867 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.252848    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b8964cf76-q7dwh"]
Sep 09 21:25:30.258965 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.258946    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"]
Sep 09 21:25:30.259296 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259284    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="storage-initializer"
Sep 09 21:25:30.259340 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259298    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="storage-initializer"
Sep 09 21:25:30.259340 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259311    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container"
Sep 09 21:25:30.259340 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259317    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container"
Sep 09 21:25:30.259340 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259325    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container"
Sep 09 21:25:30.259340 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259330    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259342    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="storage-initializer"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259347    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="storage-initializer"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259357    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kube-rbac-proxy"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259362    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kube-rbac-proxy"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259367    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kube-rbac-proxy"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259372    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kube-rbac-proxy"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259421    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kserve-container"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259428    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ca6757-691d-4cf2-a436-93de6f149ea0" containerName="kube-rbac-proxy"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259435    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kserve-container"
Sep 09 21:25:30.259502 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.259440    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" containerName="kube-rbac-proxy"
Sep 09 21:25:30.264060 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.264020    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.266248 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.266230    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:25:30.266348 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.266275    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 09 21:25:30.266494 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.266468    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 09 21:25:30.266494 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.266494    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:25:30.266701 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.266592    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:25:30.266857 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.266839    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:25:30.266937 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.266842    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:25:30.273909 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.273886    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"]
Sep 09 21:25:30.417702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.417665    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.417702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.417706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rqp\" (UniqueName: \"kubernetes.io/projected/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-api-access-z5rqp\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.417931 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.417730    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.417931 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.417838    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.417931 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.417871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.519248 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.519164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.519248 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.519203    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rqp\" (UniqueName: \"kubernetes.io/projected/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-api-access-z5rqp\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.519439 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:25:30.519321    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 21:25:30.519439 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.519330    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.519439 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:25:30.519390    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-proxy-tls podName:ad8ef79b-b5bc-4d0f-9e1f-1c663969e987 nodeName:}" failed. No retries permitted until 2026-09-09 21:25:31.019370999 +0000 UTC m=+2020.580398308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-proxy-tls") pod "isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" (UID: "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 09 21:25:30.519439 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.519422    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.519711 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.519453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.519874 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.519856    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.520011 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.519991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.520158 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.520140    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.528191 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.528169    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rqp\" (UniqueName: \"kubernetes.io/projected/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-api-access-z5rqp\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:30.935938 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:30.935907    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca4a93ef-3294-4e64-a961-16afefe14c52" path="/var/lib/kubelet/pods/ca4a93ef-3294-4e64-a961-16afefe14c52/volumes"
Sep 09 21:25:31.024609 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:31.024569    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:31.027118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:31.027094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-proxy-tls\") pod \"isvc-canary-rollback-predictor-6cb68d45bc-mqcgz\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:31.174782 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:31.174718    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:31.468995 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:31.468923    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"]
Sep 09 21:25:31.471887 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:25:31.471858    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8ef79b_b5bc_4d0f_9e1f_1c663969e987.slice/crio-e327008b41d39a0f7365020249ee829f5e4c53ee848670896c7a05ecd9770347 WatchSource:0}: Error finding container e327008b41d39a0f7365020249ee829f5e4c53ee848670896c7a05ecd9770347: Status 404 returned error can't find the container with id e327008b41d39a0f7365020249ee829f5e4c53ee848670896c7a05ecd9770347
Sep 09 21:25:32.237681 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:32.237647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" event={"ID":"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987","Type":"ContainerStarted","Data":"aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379"}
Sep 09 21:25:32.238096 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:32.237708    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" event={"ID":"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987","Type":"ContainerStarted","Data":"e327008b41d39a0f7365020249ee829f5e4c53ee848670896c7a05ecd9770347"}
Sep 09 21:25:36.258002 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:36.257971    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerID="aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379" exitCode=0
Sep 09 21:25:36.258426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:36.258035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" event={"ID":"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987","Type":"ContainerDied","Data":"aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379"}
Sep 09 21:25:37.265185 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:37.265148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" event={"ID":"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987","Type":"ContainerStarted","Data":"c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f"}
Sep 09 21:25:37.265617 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:37.265210    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" event={"ID":"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987","Type":"ContainerStarted","Data":"650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693"}
Sep 09 21:25:37.265617 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:37.265469    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:37.265736 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:37.265636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:37.266854 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:37.266828    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 09 21:25:37.287650 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:37.287602    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podStartSLOduration=3.433839234 podStartE2EDuration="7.287588044s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.85374881s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:25:37.285492313 +0000 UTC m=+2026.846519630" watchObservedRunningTime="2026-09-09 21:25:37.287588044 +0000 UTC m=+2026.848615361"
Sep 09 21:25:38.269892 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:38.269853    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 09 21:25:43.274172 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:43.274144    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:25:43.274655 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:43.274627    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 09 21:25:53.274899 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:25:53.274862    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 09 21:26:03.275320 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:26:03.275284    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 09 21:26:13.275400 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:26:13.275319    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 09 21:26:23.275001 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:26:23.274957    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 09 21:26:33.274999 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:26:33.274962    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 09 21:26:43.274719 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:26:43.274676    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 09 21:26:49.931978 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:26:49.931946    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:27:05.075162 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:27:05.075055    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:27:05.079037 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:27:05.077778    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:28:50.242554 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.242521    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"]
Sep 09 21:28:50.246076 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.246058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.249090 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.249069    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 09 21:28:50.249173 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.249070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 09 21:28:50.258216 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.258192    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"]
Sep 09 21:28:50.341245 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.341210    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjccs\" (UniqueName: \"kubernetes.io/projected/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-api-access-sjccs\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.341245 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.341246    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/082019c1-6e5c-4993-93f9-6f13a5d4adf8-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.341450 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.341274    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.341450 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.341295    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.341450 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.341311    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.442295 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.442261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjccs\" (UniqueName: \"kubernetes.io/projected/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-api-access-sjccs\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.442295 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.442298    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/082019c1-6e5c-4993-93f9-6f13a5d4adf8-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.442558 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.442327    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.442558 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.442344    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.442558 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.442360    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.442811 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.442791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.443086 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.443069    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.443145 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.443126    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.444988 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.444970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/082019c1-6e5c-4993-93f9-6f13a5d4adf8-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.453100 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.453076    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjccs\" (UniqueName: \"kubernetes.io/projected/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-api-access-sjccs\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.556823 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.556714    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:50.872129 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:50.872102    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"]
Sep 09 21:28:50.874616 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:28:50.874569    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082019c1_6e5c_4993_93f9_6f13a5d4adf8.slice/crio-e560ed1e687385f889fd39ca46cbee49f8c517477cd4f139ae06025ba16eb3c4 WatchSource:0}: Error finding container e560ed1e687385f889fd39ca46cbee49f8c517477cd4f139ae06025ba16eb3c4: Status 404 returned error can't find the container with id e560ed1e687385f889fd39ca46cbee49f8c517477cd4f139ae06025ba16eb3c4
Sep 09 21:28:51.192120 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:51.192034    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" event={"ID":"082019c1-6e5c-4993-93f9-6f13a5d4adf8","Type":"ContainerStarted","Data":"8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288"}
Sep 09 21:28:51.192120 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:51.192096    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" event={"ID":"082019c1-6e5c-4993-93f9-6f13a5d4adf8","Type":"ContainerStarted","Data":"e560ed1e687385f889fd39ca46cbee49f8c517477cd4f139ae06025ba16eb3c4"}
Sep 09 21:28:55.212783 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:55.212732    2580 generic.go:356] "Generic (PLEG): container finished" podID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerID="8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288" exitCode=0
Sep 09 21:28:55.213204 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:55.212809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" event={"ID":"082019c1-6e5c-4993-93f9-6f13a5d4adf8","Type":"ContainerDied","Data":"8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288"}
Sep 09 21:28:56.220234 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:56.220198    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" event={"ID":"082019c1-6e5c-4993-93f9-6f13a5d4adf8","Type":"ContainerStarted","Data":"cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b"}
Sep 09 21:28:56.220644 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:56.220265    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" event={"ID":"082019c1-6e5c-4993-93f9-6f13a5d4adf8","Type":"ContainerStarted","Data":"c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1"}
Sep 09 21:28:56.220644 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:56.220473    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:28:56.241917 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:56.241874    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" podStartSLOduration=2.479697132 podStartE2EDuration="6.241855315s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.762158183s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:28:56.240938445 +0000 UTC m=+2225.801965761" watchObservedRunningTime="2026-09-09 21:28:56.241855315 +0000 UTC m=+2225.802882632"
Sep 09 21:28:57.225196 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:28:57.225166    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:29:03.235489 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:03.235454    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:29:20.388679 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:20.388599    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"]
Sep 09 21:29:20.389095 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:20.388898    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container" containerID="cri-o://650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693" gracePeriod=30
Sep 09 21:29:20.389095 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:20.389014    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kube-rbac-proxy" containerID="cri-o://c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f" gracePeriod=30
Sep 09 21:29:21.341226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:21.341193    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerID="c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f" exitCode=2
Sep 09 21:29:21.341413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:21.341231    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" event={"ID":"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987","Type":"ContainerDied","Data":"c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f"}
Sep 09 21:29:23.271125 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:23.271084    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 09 21:29:25.536589 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.536557    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:29:25.632659 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.632576    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5rqp\" (UniqueName: \"kubernetes.io/projected/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-api-access-z5rqp\") pod \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") "
Sep 09 21:29:25.632659 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.632618    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-rbac-proxy-sar-config\") pod \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") "
Sep 09 21:29:25.632924 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.632662    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-proxy-tls\") pod \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") "
Sep 09 21:29:25.632924 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.632684    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-cabundle-cert\") pod \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") "
Sep 09 21:29:25.632924 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.632713    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kserve-provision-location\") pod \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\" (UID: \"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987\") "
Sep 09 21:29:25.633129 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.633094    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" (UID: "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:25.633129 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.633116    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" (UID: "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:29:25.633238 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.633102    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" (UID: "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:29:25.634996 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.634973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" (UID: "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:29:25.635098 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.635082    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-api-access-z5rqp" (OuterVolumeSpecName: "kube-api-access-z5rqp") pod "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" (UID: "ad8ef79b-b5bc-4d0f-9e1f-1c663969e987"). InnerVolumeSpecName "kube-api-access-z5rqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:29:25.734146 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.734113    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5rqp\" (UniqueName: \"kubernetes.io/projected/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-api-access-z5rqp\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.734146 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.734143    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.734146 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.734154    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.734458 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.734167    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.734458 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.734179    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:25.813525 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.813483    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"]
Sep 09 21:29:25.813949 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.813930    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="storage-initializer"
Sep 09 21:29:25.814048 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.813951    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="storage-initializer"
Sep 09 21:29:25.814048 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.813971    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container"
Sep 09 21:29:25.814048 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.813979    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container"
Sep 09 21:29:25.814048 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.813998    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kube-rbac-proxy"
Sep 09 21:29:25.814048 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.814007    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kube-rbac-proxy"
Sep 09 21:29:25.814310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.814094    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kube-rbac-proxy"
Sep 09 21:29:25.814310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.814109    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerName="kserve-container"
Sep 09 21:29:25.817575 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.817556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:25.820908 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.820889    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 09 21:29:25.821949 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.821932    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 09 21:29:25.831700 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.831675    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"]
Sep 09 21:29:25.935413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.935327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:25.935413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.935384    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:25.935413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.935410    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:25.935631 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.935455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9x4s\" (UniqueName: \"kubernetes.io/projected/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-api-access-b9x4s\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:25.935631 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:25.935502    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.036403 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.036369    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.036570 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.036438    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.036570 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.036496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.036570 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.036528    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.036570 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.036556    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9x4s\" (UniqueName: \"kubernetes.io/projected/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-api-access-b9x4s\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.037033 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.037005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.037161 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.037139    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-cabundle-cert\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.037257 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.037238    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.039151 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.039130    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-proxy-tls\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.044884 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.044867    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9x4s\" (UniqueName: \"kubernetes.io/projected/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-api-access-b9x4s\") pod \"isvc-canary-stop-predictor-59fbfd5bc-qqhff\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.128699 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.128660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:26.369040 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.368997    2580 generic.go:356] "Generic (PLEG): container finished" podID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" containerID="650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693" exitCode=0
Sep 09 21:29:26.369226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.369054    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"
Sep 09 21:29:26.369226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.369084    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" event={"ID":"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987","Type":"ContainerDied","Data":"650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693"}
Sep 09 21:29:26.369226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.369146    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz" event={"ID":"ad8ef79b-b5bc-4d0f-9e1f-1c663969e987","Type":"ContainerDied","Data":"e327008b41d39a0f7365020249ee829f5e4c53ee848670896c7a05ecd9770347"}
Sep 09 21:29:26.369226 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.369176    2580 scope.go:117] "RemoveContainer" containerID="c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f"
Sep 09 21:29:26.378619 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.378602    2580 scope.go:117] "RemoveContainer" containerID="650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693"
Sep 09 21:29:26.386969 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.386951    2580 scope.go:117] "RemoveContainer" containerID="aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379"
Sep 09 21:29:26.392678 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.392653    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"]
Sep 09 21:29:26.395996 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.395973    2580 scope.go:117] "RemoveContainer" containerID="c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f"
Sep 09 21:29:26.396347 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:29:26.396325    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f\": container with ID starting with c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f not found: ID does not exist" containerID="c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f"
Sep 09 21:29:26.396426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.396355    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f"} err="failed to get container status \"c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f\": rpc error: code = NotFound desc = could not find container \"c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f\": container with ID starting with c7c44e245c7057819b2023e8a2e48b057d647decd44aa08d8eb38ec30974510f not found: ID does not exist"
Sep 09 21:29:26.396426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.396371    2580 scope.go:117] "RemoveContainer" containerID="650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693"
Sep 09 21:29:26.396672 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:29:26.396653    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693\": container with ID starting with 650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693 not found: ID does not exist" containerID="650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693"
Sep 09 21:29:26.396750 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.396685    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693"} err="failed to get container status \"650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693\": rpc error: code = NotFound desc = could not find container \"650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693\": container with ID starting with 650f76d3ef3035fe1762944d34a0640d8f3c629145621d784466c4c8a5a24693 not found: ID does not exist"
Sep 09 21:29:26.396750 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.396709    2580 scope.go:117] "RemoveContainer" containerID="aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379"
Sep 09 21:29:26.397022 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:29:26.397004    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379\": container with ID starting with aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379 not found: ID does not exist" containerID="aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379"
Sep 09 21:29:26.397068 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.397026    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379"} err="failed to get container status \"aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379\": rpc error: code = NotFound desc = could not find container \"aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379\": container with ID starting with aff91c942706159b630a2c9555b8ce38d2395c67805eb439b2d4429dfa460379 not found: ID does not exist"
Sep 09 21:29:26.397854 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.397835    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6cb68d45bc-mqcgz"]
Sep 09 21:29:26.410512 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.410313    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"]
Sep 09 21:29:26.413019 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:29:26.412997    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff6c024_0a07_40b8_92d8_bedf3191cdf9.slice/crio-65a4bbe8d96851c22ca14e02738aeadc75bb2ee1c3c24b551dd9324de915368e WatchSource:0}: Error finding container 65a4bbe8d96851c22ca14e02738aeadc75bb2ee1c3c24b551dd9324de915368e: Status 404 returned error can't find the container with id 65a4bbe8d96851c22ca14e02738aeadc75bb2ee1c3c24b551dd9324de915368e
Sep 09 21:29:26.936316 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:26.936272    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad8ef79b-b5bc-4d0f-9e1f-1c663969e987" path="/var/lib/kubelet/pods/ad8ef79b-b5bc-4d0f-9e1f-1c663969e987/volumes"
Sep 09 21:29:27.375778 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:27.375726    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" event={"ID":"9ff6c024-0a07-40b8-92d8-bedf3191cdf9","Type":"ContainerStarted","Data":"de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f"}
Sep 09 21:29:27.375992 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:27.375828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" event={"ID":"9ff6c024-0a07-40b8-92d8-bedf3191cdf9","Type":"ContainerStarted","Data":"65a4bbe8d96851c22ca14e02738aeadc75bb2ee1c3c24b551dd9324de915368e"}
Sep 09 21:29:31.398814 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:31.398752    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerID="de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f" exitCode=0
Sep 09 21:29:31.399260 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:31.398832    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" event={"ID":"9ff6c024-0a07-40b8-92d8-bedf3191cdf9","Type":"ContainerDied","Data":"de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f"}
Sep 09 21:29:32.407076 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:32.407040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" event={"ID":"9ff6c024-0a07-40b8-92d8-bedf3191cdf9","Type":"ContainerStarted","Data":"91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13"}
Sep 09 21:29:32.407464 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:32.407101    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" event={"ID":"9ff6c024-0a07-40b8-92d8-bedf3191cdf9","Type":"ContainerStarted","Data":"201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe"}
Sep 09 21:29:32.407464 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:32.407365    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:32.407464 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:32.407408    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:32.408884 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:32.408854    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:29:33.239693 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:33.239663    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:29:33.262957 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:33.262908    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podStartSLOduration=4.366644825 podStartE2EDuration="8.262891065s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.89624624s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:29:32.44039971 +0000 UTC m=+2262.001427028" watchObservedRunningTime="2026-09-09 21:29:33.262891065 +0000 UTC m=+2262.823918378"
Sep 09 21:29:33.411842 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:33.411802    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:29:38.416183 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:38.416150    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:29:38.416649 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:38.416623    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:29:40.292435 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:40.292402    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"]
Sep 09 21:29:40.292847 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:40.292648    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kserve-container" containerID="cri-o://c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1" gracePeriod=30
Sep 09 21:29:40.292847 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:40.292673    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kube-rbac-proxy" containerID="cri-o://cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b" gracePeriod=30
Sep 09 21:29:40.447523 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:40.447487    2580 generic.go:356] "Generic (PLEG): container finished" podID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerID="cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b" exitCode=2
Sep 09 21:29:40.447685 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:40.447562    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" event={"ID":"082019c1-6e5c-4993-93f9-6f13a5d4adf8","Type":"ContainerDied","Data":"cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b"}
Sep 09 21:29:43.231006 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:43.230967    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 09 21:29:43.236049 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:43.236021    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.57:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 09 21:29:45.479819 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.479774    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"]
Sep 09 21:29:45.483707 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.483689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.485774 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.485744    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 09 21:29:45.485871 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.485744    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 09 21:29:45.493191 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.493168    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"]
Sep 09 21:29:45.608203 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.608169    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.608386 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.608252    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d3e95e-d03f-4abe-844d-e5c7ba225755-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.608386 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.608288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8c5\" (UniqueName: \"kubernetes.io/projected/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-api-access-dn8c5\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.608386 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.608303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.608386 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.608328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.709011 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.708973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d3e95e-d03f-4abe-844d-e5c7ba225755-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.709224 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.709023    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8c5\" (UniqueName: \"kubernetes.io/projected/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-api-access-dn8c5\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.709224 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.709043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.709224 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.709060    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.709224 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.709094    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.709571 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.709544    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.709920 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.709899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.709920 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.709912    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.711702 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.711673    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d3e95e-d03f-4abe-844d-e5c7ba225755-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.717516 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.717493    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8c5\" (UniqueName: \"kubernetes.io/projected/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-api-access-dn8c5\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:45.794950 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:45.794857    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:46.070814 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:46.070782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"]
Sep 09 21:29:46.071781 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:29:46.071737    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d3e95e_d03f_4abe_844d_e5c7ba225755.slice/crio-d567e7a234228b0a857fe3839595a1733da9b85bd970744e5c994e11279aca85 WatchSource:0}: Error finding container d567e7a234228b0a857fe3839595a1733da9b85bd970744e5c994e11279aca85: Status 404 returned error can't find the container with id d567e7a234228b0a857fe3839595a1733da9b85bd970744e5c994e11279aca85
Sep 09 21:29:46.479351 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:46.479305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" event={"ID":"b6d3e95e-d03f-4abe-844d-e5c7ba225755","Type":"ContainerStarted","Data":"9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda"}
Sep 09 21:29:46.479561 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:46.479383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" event={"ID":"b6d3e95e-d03f-4abe-844d-e5c7ba225755","Type":"ContainerStarted","Data":"d567e7a234228b0a857fe3839595a1733da9b85bd970744e5c994e11279aca85"}
Sep 09 21:29:46.944236 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:46.944208    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:29:47.120710 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.120683    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kserve-provision-location\") pod \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") "
Sep 09 21:29:47.120919 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.120732    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/082019c1-6e5c-4993-93f9-6f13a5d4adf8-proxy-tls\") pod \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") "
Sep 09 21:29:47.120919 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.120778    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-rbac-proxy-sar-config\") pod \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") "
Sep 09 21:29:47.121041 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.120912    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjccs\" (UniqueName: \"kubernetes.io/projected/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-api-access-sjccs\") pod \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") "
Sep 09 21:29:47.121041 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.120951    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-cabundle-cert\") pod \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\" (UID: \"082019c1-6e5c-4993-93f9-6f13a5d4adf8\") "
Sep 09 21:29:47.121138 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.121073    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "082019c1-6e5c-4993-93f9-6f13a5d4adf8" (UID: "082019c1-6e5c-4993-93f9-6f13a5d4adf8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:29:47.121138 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.121083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "082019c1-6e5c-4993-93f9-6f13a5d4adf8" (UID: "082019c1-6e5c-4993-93f9-6f13a5d4adf8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:29:47.121290 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.121272    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:47.121345 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.121296    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:47.121345 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.121319    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "082019c1-6e5c-4993-93f9-6f13a5d4adf8" (UID: "082019c1-6e5c-4993-93f9-6f13a5d4adf8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:29:47.123084 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.123064    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082019c1-6e5c-4993-93f9-6f13a5d4adf8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "082019c1-6e5c-4993-93f9-6f13a5d4adf8" (UID: "082019c1-6e5c-4993-93f9-6f13a5d4adf8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:29:47.123147 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.123097    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-api-access-sjccs" (OuterVolumeSpecName: "kube-api-access-sjccs") pod "082019c1-6e5c-4993-93f9-6f13a5d4adf8" (UID: "082019c1-6e5c-4993-93f9-6f13a5d4adf8"). InnerVolumeSpecName "kube-api-access-sjccs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:29:47.221940 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.221908    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/082019c1-6e5c-4993-93f9-6f13a5d4adf8-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:47.221940 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.221936    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjccs\" (UniqueName: \"kubernetes.io/projected/082019c1-6e5c-4993-93f9-6f13a5d4adf8-kube-api-access-sjccs\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:47.221940 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.221945    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/082019c1-6e5c-4993-93f9-6f13a5d4adf8-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:29:47.486819 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.486720    2580 generic.go:356] "Generic (PLEG): container finished" podID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerID="c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1" exitCode=0
Sep 09 21:29:47.486819 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.486809    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" event={"ID":"082019c1-6e5c-4993-93f9-6f13a5d4adf8","Type":"ContainerDied","Data":"c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1"}
Sep 09 21:29:47.487016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.486815    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"
Sep 09 21:29:47.487016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.486873    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm" event={"ID":"082019c1-6e5c-4993-93f9-6f13a5d4adf8","Type":"ContainerDied","Data":"e560ed1e687385f889fd39ca46cbee49f8c517477cd4f139ae06025ba16eb3c4"}
Sep 09 21:29:47.487016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.486913    2580 scope.go:117] "RemoveContainer" containerID="cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b"
Sep 09 21:29:47.496732 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.496711    2580 scope.go:117] "RemoveContainer" containerID="c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1"
Sep 09 21:29:47.506771 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.506732    2580 scope.go:117] "RemoveContainer" containerID="8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288"
Sep 09 21:29:47.514868 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.514846    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"]
Sep 09 21:29:47.516058 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.516042    2580 scope.go:117] "RemoveContainer" containerID="cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b"
Sep 09 21:29:47.516338 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:29:47.516313    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b\": container with ID starting with cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b not found: ID does not exist" containerID="cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b"
Sep 09 21:29:47.516468 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.516342    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b"} err="failed to get container status \"cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b\": rpc error: code = NotFound desc = could not find container \"cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b\": container with ID starting with cb8338275b75d2e2ad43682323d1a1fcfd847aa0a1fe983eeabc6146eabeb44b not found: ID does not exist"
Sep 09 21:29:47.516468 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.516360    2580 scope.go:117] "RemoveContainer" containerID="c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1"
Sep 09 21:29:47.516683 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:29:47.516656    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1\": container with ID starting with c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1 not found: ID does not exist" containerID="c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1"
Sep 09 21:29:47.516830 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.516713    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1"} err="failed to get container status \"c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1\": rpc error: code = NotFound desc = could not find container \"c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1\": container with ID starting with c31887366a8ea1fdea123ec4d639263aabe5692d7cbfa6c21fbc68bd92f5d3d1 not found: ID does not exist"
Sep 09 21:29:47.516830 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.516738    2580 scope.go:117] "RemoveContainer" containerID="8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288"
Sep 09 21:29:47.517098 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:29:47.517069    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288\": container with ID starting with 8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288 not found: ID does not exist" containerID="8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288"
Sep 09 21:29:47.517222 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.517111    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288"} err="failed to get container status \"8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288\": rpc error: code = NotFound desc = could not find container \"8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288\": container with ID starting with 8c52d458caf461fce6e8a0645b16ea27b7e9fd155578c08b4460d9a3e4f8c288 not found: ID does not exist"
Sep 09 21:29:47.519419 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:47.519399    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-zk7gm"]
Sep 09 21:29:47.526554 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:29:47.526534    2580 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod082019c1_6e5c_4993_93f9_6f13a5d4adf8.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 21:29:48.417018 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:48.416984    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:29:48.935820 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:48.935782    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" path="/var/lib/kubelet/pods/082019c1-6e5c-4993-93f9-6f13a5d4adf8/volumes"
Sep 09 21:29:50.504804 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:50.504744    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerID="9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda" exitCode=0
Sep 09 21:29:50.505173 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:50.504819    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" event={"ID":"b6d3e95e-d03f-4abe-844d-e5c7ba225755","Type":"ContainerDied","Data":"9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda"}
Sep 09 21:29:51.512195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:51.512108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" event={"ID":"b6d3e95e-d03f-4abe-844d-e5c7ba225755","Type":"ContainerStarted","Data":"af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab"}
Sep 09 21:29:51.512195 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:51.512175    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" event={"ID":"b6d3e95e-d03f-4abe-844d-e5c7ba225755","Type":"ContainerStarted","Data":"63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f"}
Sep 09 21:29:51.512612 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:51.512376    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:51.535563 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:51.535511    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" podStartSLOduration=2.5772083759999997 podStartE2EDuration="6.535497131s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.958288755s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:29:51.533550888 +0000 UTC m=+2281.094578207" watchObservedRunningTime="2026-09-09 21:29:51.535497131 +0000 UTC m=+2281.096524448"
Sep 09 21:29:52.516794 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:52.516748    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:29:58.416907 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:58.416867    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:29:58.526415 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:29:58.526380    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:30:08.417415 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:08.417373    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:30:18.417165 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:18.417123    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:30:28.416688 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:28.416644    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:30:28.531562 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:28.531533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:30:35.719736 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:35.719700    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"]
Sep 09 21:30:35.720266 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:35.720006    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kserve-container" containerID="cri-o://63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f" gracePeriod=30
Sep 09 21:30:35.720266 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:35.720087    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kube-rbac-proxy" containerID="cri-o://af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab" gracePeriod=30
Sep 09 21:30:36.736297 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:36.736264    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerID="af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab" exitCode=2
Sep 09 21:30:36.736701 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:36.736341    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" event={"ID":"b6d3e95e-d03f-4abe-844d-e5c7ba225755","Type":"ContainerDied","Data":"af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab"}
Sep 09 21:30:38.416813 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:38.416773    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:30:38.522000 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:38.521959    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 09 21:30:38.527201 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:38.527167    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.59:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 09 21:30:42.469552 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.469519    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:30:42.552103 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.552005    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-rbac-proxy-sar-config\") pod \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") "
Sep 09 21:30:42.552103 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.552074    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kserve-provision-location\") pod \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") "
Sep 09 21:30:42.552336 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.552121    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn8c5\" (UniqueName: \"kubernetes.io/projected/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-api-access-dn8c5\") pod \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") "
Sep 09 21:30:42.552336 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.552168    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d3e95e-d03f-4abe-844d-e5c7ba225755-proxy-tls\") pod \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") "
Sep 09 21:30:42.552336 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.552223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-cabundle-cert\") pod \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\" (UID: \"b6d3e95e-d03f-4abe-844d-e5c7ba225755\") "
Sep 09 21:30:42.552506 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.552428    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6d3e95e-d03f-4abe-844d-e5c7ba225755" (UID: "b6d3e95e-d03f-4abe-844d-e5c7ba225755"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:30:42.552576 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.552428    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6d3e95e-d03f-4abe-844d-e5c7ba225755" (UID: "b6d3e95e-d03f-4abe-844d-e5c7ba225755"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:30:42.552644 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.552616    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6d3e95e-d03f-4abe-844d-e5c7ba225755" (UID: "b6d3e95e-d03f-4abe-844d-e5c7ba225755"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:30:42.554487 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.554457    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6d3e95e-d03f-4abe-844d-e5c7ba225755-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6d3e95e-d03f-4abe-844d-e5c7ba225755" (UID: "b6d3e95e-d03f-4abe-844d-e5c7ba225755"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:30:42.554593 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.554495    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-api-access-dn8c5" (OuterVolumeSpecName: "kube-api-access-dn8c5") pod "b6d3e95e-d03f-4abe-844d-e5c7ba225755" (UID: "b6d3e95e-d03f-4abe-844d-e5c7ba225755"). InnerVolumeSpecName "kube-api-access-dn8c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:30:42.652912 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.652874    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6d3e95e-d03f-4abe-844d-e5c7ba225755-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:30:42.652912 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.652905    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:30:42.652912 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.652917    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:30:42.653152 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.652926    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:30:42.653152 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.652934    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn8c5\" (UniqueName: \"kubernetes.io/projected/b6d3e95e-d03f-4abe-844d-e5c7ba225755-kube-api-access-dn8c5\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:30:42.772084 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.772048    2580 generic.go:356] "Generic (PLEG): container finished" podID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerID="63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f" exitCode=0
Sep 09 21:30:42.772276 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.772121    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"
Sep 09 21:30:42.772276 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.772136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" event={"ID":"b6d3e95e-d03f-4abe-844d-e5c7ba225755","Type":"ContainerDied","Data":"63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f"}
Sep 09 21:30:42.772276 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.772227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g" event={"ID":"b6d3e95e-d03f-4abe-844d-e5c7ba225755","Type":"ContainerDied","Data":"d567e7a234228b0a857fe3839595a1733da9b85bd970744e5c994e11279aca85"}
Sep 09 21:30:42.772276 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.772256    2580 scope.go:117] "RemoveContainer" containerID="af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab"
Sep 09 21:30:42.781330 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.781314    2580 scope.go:117] "RemoveContainer" containerID="63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f"
Sep 09 21:30:42.789625 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.789603    2580 scope.go:117] "RemoveContainer" containerID="9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda"
Sep 09 21:30:42.796281 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.796252    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"]
Sep 09 21:30:42.798041 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.798020    2580 scope.go:117] "RemoveContainer" containerID="af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab"
Sep 09 21:30:42.798323 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:30:42.798304    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab\": container with ID starting with af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab not found: ID does not exist" containerID="af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab"
Sep 09 21:30:42.798373 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.798332    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab"} err="failed to get container status \"af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab\": rpc error: code = NotFound desc = could not find container \"af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab\": container with ID starting with af3075a75b248554261eb6efb3654416923276321661614e1b21832ad84095ab not found: ID does not exist"
Sep 09 21:30:42.798373 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.798354    2580 scope.go:117] "RemoveContainer" containerID="63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f"
Sep 09 21:30:42.798592 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:30:42.798579    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f\": container with ID starting with 63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f not found: ID does not exist" containerID="63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f"
Sep 09 21:30:42.798629 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.798595    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f"} err="failed to get container status \"63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f\": rpc error: code = NotFound desc = could not find container \"63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f\": container with ID starting with 63f6a5c02e3801a55f5a4e9d2d0d65ee8d8cd64f471ab8863a958283a663964f not found: ID does not exist"
Sep 09 21:30:42.798629 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.798613    2580 scope.go:117] "RemoveContainer" containerID="9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda"
Sep 09 21:30:42.798836 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:30:42.798818    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda\": container with ID starting with 9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda not found: ID does not exist" containerID="9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda"
Sep 09 21:30:42.798888 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.798843    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda"} err="failed to get container status \"9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda\": rpc error: code = NotFound desc = could not find container \"9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda\": container with ID starting with 9be1a9c3c3fb9867d583944cda703a60a5a4c7e9a3eb58fafd46d465f3a3aeda not found: ID does not exist"
Sep 09 21:30:42.804633 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.804581    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-hwh9g"]
Sep 09 21:30:42.936589 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:42.936560    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" path="/var/lib/kubelet/pods/b6d3e95e-d03f-4abe-844d-e5c7ba225755/volumes"
Sep 09 21:30:48.417961 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:48.417928    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:30:55.662148 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:55.662098    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"]
Sep 09 21:30:55.662596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:55.662357    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" containerID="cri-o://201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe" gracePeriod=30
Sep 09 21:30:55.662596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:55.662402    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kube-rbac-proxy" containerID="cri-o://91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13" gracePeriod=30
Sep 09 21:30:55.838870 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:55.838829    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerID="91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13" exitCode=2
Sep 09 21:30:55.839039 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:55.838904    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" event={"ID":"9ff6c024-0a07-40b8-92d8-bedf3191cdf9","Type":"ContainerDied","Data":"91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13"}
Sep 09 21:30:58.413075 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:58.413034    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.58:8643/healthz\": dial tcp 10.132.0.58:8643: connect: connection refused"
Sep 09 21:30:58.417395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:30:58.417371    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.58:8080: connect: connection refused"
Sep 09 21:31:00.514459 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.514436    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:31:00.618869 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.618792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kserve-provision-location\") pod \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") "
Sep 09 21:31:00.619036 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.618875    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-rbac-proxy-sar-config\") pod \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") "
Sep 09 21:31:00.619036 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.618894    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-proxy-tls\") pod \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") "
Sep 09 21:31:00.619036 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.618921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-cabundle-cert\") pod \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") "
Sep 09 21:31:00.619036 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.618946    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9x4s\" (UniqueName: \"kubernetes.io/projected/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-api-access-b9x4s\") pod \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\" (UID: \"9ff6c024-0a07-40b8-92d8-bedf3191cdf9\") "
Sep 09 21:31:00.619264 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.619146    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9ff6c024-0a07-40b8-92d8-bedf3191cdf9" (UID: "9ff6c024-0a07-40b8-92d8-bedf3191cdf9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:31:00.619264 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.619245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ff6c024-0a07-40b8-92d8-bedf3191cdf9" (UID: "9ff6c024-0a07-40b8-92d8-bedf3191cdf9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:31:00.619378 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.619340    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9ff6c024-0a07-40b8-92d8-bedf3191cdf9" (UID: "9ff6c024-0a07-40b8-92d8-bedf3191cdf9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:31:00.621345 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.621311    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ff6c024-0a07-40b8-92d8-bedf3191cdf9" (UID: "9ff6c024-0a07-40b8-92d8-bedf3191cdf9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:31:00.621445 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.621373    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-api-access-b9x4s" (OuterVolumeSpecName: "kube-api-access-b9x4s") pod "9ff6c024-0a07-40b8-92d8-bedf3191cdf9" (UID: "9ff6c024-0a07-40b8-92d8-bedf3191cdf9"). InnerVolumeSpecName "kube-api-access-b9x4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:31:00.719783 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.719728    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:31:00.719783 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.719786    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:31:00.720007 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.719798    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:31:00.720007 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.719814    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9x4s\" (UniqueName: \"kubernetes.io/projected/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kube-api-access-b9x4s\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:31:00.720007 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.719822    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9ff6c024-0a07-40b8-92d8-bedf3191cdf9-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:31:00.864335 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.864299    2580 generic.go:356] "Generic (PLEG): container finished" podID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerID="201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe" exitCode=0
Sep 09 21:31:00.864520 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.864355    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"
Sep 09 21:31:00.864520 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.864357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" event={"ID":"9ff6c024-0a07-40b8-92d8-bedf3191cdf9","Type":"ContainerDied","Data":"201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe"}
Sep 09 21:31:00.864520 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.864437    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff" event={"ID":"9ff6c024-0a07-40b8-92d8-bedf3191cdf9","Type":"ContainerDied","Data":"65a4bbe8d96851c22ca14e02738aeadc75bb2ee1c3c24b551dd9324de915368e"}
Sep 09 21:31:00.864520 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.864473    2580 scope.go:117] "RemoveContainer" containerID="91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13"
Sep 09 21:31:00.874800 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.874552    2580 scope.go:117] "RemoveContainer" containerID="201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe"
Sep 09 21:31:00.883048 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.883025    2580 scope.go:117] "RemoveContainer" containerID="de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f"
Sep 09 21:31:00.893657 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.893142    2580 scope.go:117] "RemoveContainer" containerID="91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13"
Sep 09 21:31:00.893657 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.893242    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"]
Sep 09 21:31:00.894014 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:31:00.893983    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13\": container with ID starting with 91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13 not found: ID does not exist" containerID="91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13"
Sep 09 21:31:00.894101 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.894026    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13"} err="failed to get container status \"91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13\": rpc error: code = NotFound desc = could not find container \"91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13\": container with ID starting with 91721d333effe8040ca6aa501839fec0b770061b40c14f994da066f147df4b13 not found: ID does not exist"
Sep 09 21:31:00.894101 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.894051    2580 scope.go:117] "RemoveContainer" containerID="201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe"
Sep 09 21:31:00.895003 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:31:00.894970    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe\": container with ID starting with 201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe not found: ID does not exist" containerID="201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe"
Sep 09 21:31:00.895101 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.895011    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe"} err="failed to get container status \"201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe\": rpc error: code = NotFound desc = could not find container \"201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe\": container with ID starting with 201f99997c95deb694294a84def0156c511025f1437d2a6db78cf00e817a28fe not found: ID does not exist"
Sep 09 21:31:00.895101 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.895035    2580 scope.go:117] "RemoveContainer" containerID="de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f"
Sep 09 21:31:00.895367 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:31:00.895340    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f\": container with ID starting with de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f not found: ID does not exist" containerID="de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f"
Sep 09 21:31:00.895418 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.895375    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f"} err="failed to get container status \"de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f\": rpc error: code = NotFound desc = could not find container \"de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f\": container with ID starting with de72bbfb5136f9e32115772f8c3010ac11a6f2d251797375b792c7a1210ede5f not found: ID does not exist"
Sep 09 21:31:00.895603 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.895584    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-59fbfd5bc-qqhff"]
Sep 09 21:31:00.935445 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:31:00.935416    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" path="/var/lib/kubelet/pods/9ff6c024-0a07-40b8-92d8-bedf3191cdf9/volumes"
Sep 09 21:32:05.105893 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:05.105864    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:32:05.108664 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:05.108646    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:32:06.162016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.161980    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"]
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162318    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162329    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162338    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kube-rbac-proxy"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162344    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kube-rbac-proxy"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162352    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="storage-initializer"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162358    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="storage-initializer"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162365    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kserve-container"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162370    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kserve-container"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162378    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kube-rbac-proxy"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162383    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kube-rbac-proxy"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162391    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="storage-initializer"
Sep 09 21:32:06.162395 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162396    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="storage-initializer"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162404    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kserve-container"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162409    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kserve-container"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162417    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="storage-initializer"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162422    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="storage-initializer"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162454    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kube-rbac-proxy"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162459    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kube-rbac-proxy"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162510    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kube-rbac-proxy"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162517    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ff6c024-0a07-40b8-92d8-bedf3191cdf9" containerName="kserve-container"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162526    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kube-rbac-proxy"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162532    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6d3e95e-d03f-4abe-844d-e5c7ba225755" containerName="kserve-container"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162537    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kserve-container"
Sep 09 21:32:06.162806 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.162542    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="082019c1-6e5c-4993-93f9-6f13a5d4adf8" containerName="kube-rbac-proxy"
Sep 09 21:32:06.165510 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.165494    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.168016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.167990    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 09 21:32:06.168016 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.168006    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:32:06.168237 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.168027    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:32:06.168237 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.168205    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:32:06.168402 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.168383    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:32:06.168770 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.168744    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:32:06.170425 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.170406    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:32:06.180149 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.180126    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"]
Sep 09 21:32:06.259998 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.259957    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.260197 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.260013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409b00a9-abfa-46a7-b432-3f428d5b05c0-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.260197 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.260064    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68bp\" (UniqueName: \"kubernetes.io/projected/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-api-access-n68bp\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.260197 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.260099    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.260197 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.260154    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409b00a9-abfa-46a7-b432-3f428d5b05c0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.361006 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.360968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409b00a9-abfa-46a7-b432-3f428d5b05c0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.361006 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.361014    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.361285 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.361059    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409b00a9-abfa-46a7-b432-3f428d5b05c0-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.361285 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.361092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68bp\" (UniqueName: \"kubernetes.io/projected/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-api-access-n68bp\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.361285 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:32:06.361133    2580 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 09 21:32:06.361285 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:32:06.361214    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/409b00a9-abfa-46a7-b432-3f428d5b05c0-proxy-tls podName:409b00a9-abfa-46a7-b432-3f428d5b05c0 nodeName:}" failed. No retries permitted until 2026-09-09 21:32:06.86119404 +0000 UTC m=+2416.422221339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/409b00a9-abfa-46a7-b432-3f428d5b05c0-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" (UID: "409b00a9-abfa-46a7-b432-3f428d5b05c0") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Sep 09 21:32:06.361285 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.361140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.361551 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.361535    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409b00a9-abfa-46a7-b432-3f428d5b05c0-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.361888 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.361866    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.361997 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.361943    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.372222 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.372200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68bp\" (UniqueName: \"kubernetes.io/projected/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-api-access-n68bp\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.864642 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.864598    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409b00a9-abfa-46a7-b432-3f428d5b05c0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:06.867343 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:06.867319    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409b00a9-abfa-46a7-b432-3f428d5b05c0-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:07.076706 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:07.076666    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:07.356536 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:07.356500    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"]
Sep 09 21:32:07.358965 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:32:07.358932    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409b00a9_abfa_46a7_b432_3f428d5b05c0.slice/crio-eb593fc501179389a68efc6a8822de8a5ad453c6673c151ecb4043ac5e3efef0 WatchSource:0}: Error finding container eb593fc501179389a68efc6a8822de8a5ad453c6673c151ecb4043ac5e3efef0: Status 404 returned error can't find the container with id eb593fc501179389a68efc6a8822de8a5ad453c6673c151ecb4043ac5e3efef0
Sep 09 21:32:08.188100 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:08.188058    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" event={"ID":"409b00a9-abfa-46a7-b432-3f428d5b05c0","Type":"ContainerStarted","Data":"238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b"}
Sep 09 21:32:08.188322 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:08.188166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" event={"ID":"409b00a9-abfa-46a7-b432-3f428d5b05c0","Type":"ContainerStarted","Data":"eb593fc501179389a68efc6a8822de8a5ad453c6673c151ecb4043ac5e3efef0"}
Sep 09 21:32:12.209460 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:12.209428    2580 generic.go:356] "Generic (PLEG): container finished" podID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerID="238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b" exitCode=0
Sep 09 21:32:12.209861 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:12.209503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" event={"ID":"409b00a9-abfa-46a7-b432-3f428d5b05c0","Type":"ContainerDied","Data":"238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b"}
Sep 09 21:32:13.216388 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:13.216354    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" event={"ID":"409b00a9-abfa-46a7-b432-3f428d5b05c0","Type":"ContainerStarted","Data":"ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd"}
Sep 09 21:32:13.216753 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:13.216415    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" event={"ID":"409b00a9-abfa-46a7-b432-3f428d5b05c0","Type":"ContainerStarted","Data":"6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f"}
Sep 09 21:32:13.216753 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:13.216607    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:13.216753 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:13.216659    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:13.249599 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:13.249547    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" podStartSLOduration=3.033537723 podStartE2EDuration="7.249531023s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.2159933s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:32:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:32:13.248461981 +0000 UTC m=+2422.809489300" watchObservedRunningTime="2026-09-09 21:32:13.249531023 +0000 UTC m=+2422.810558342"
Sep 09 21:32:19.227322 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:19.227295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:32:49.253119 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:49.253058    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 09 21:32:59.230544 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:32:59.230512    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:33:06.242879 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:06.242844    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"]
Sep 09 21:33:06.243282 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:06.243093    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kserve-container" containerID="cri-o://6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f" gracePeriod=30
Sep 09 21:33:06.243282 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:06.243138    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kube-rbac-proxy" containerID="cri-o://ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd" gracePeriod=30
Sep 09 21:33:06.476134 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:06.476095    2580 generic.go:356] "Generic (PLEG): container finished" podID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerID="ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd" exitCode=2
Sep 09 21:33:06.476302 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:06.476172    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" event={"ID":"409b00a9-abfa-46a7-b432-3f428d5b05c0","Type":"ContainerDied","Data":"ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd"}
Sep 09 21:33:09.222062 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:09.222024    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Sep 09 21:33:13.799644 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.799617    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:33:13.818942 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.818881    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-cabundle-cert\") pod \"409b00a9-abfa-46a7-b432-3f428d5b05c0\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") "
Sep 09 21:33:13.818942 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.818916    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409b00a9-abfa-46a7-b432-3f428d5b05c0-kserve-provision-location\") pod \"409b00a9-abfa-46a7-b432-3f428d5b05c0\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") "
Sep 09 21:33:13.819157 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.818957    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409b00a9-abfa-46a7-b432-3f428d5b05c0-proxy-tls\") pod \"409b00a9-abfa-46a7-b432-3f428d5b05c0\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") "
Sep 09 21:33:13.819157 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.818985    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n68bp\" (UniqueName: \"kubernetes.io/projected/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-api-access-n68bp\") pod \"409b00a9-abfa-46a7-b432-3f428d5b05c0\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") "
Sep 09 21:33:13.819157 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.818999    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-rbac-proxy-sar-config\") pod \"409b00a9-abfa-46a7-b432-3f428d5b05c0\" (UID: \"409b00a9-abfa-46a7-b432-3f428d5b05c0\") "
Sep 09 21:33:13.819311 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.819220    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "409b00a9-abfa-46a7-b432-3f428d5b05c0" (UID: "409b00a9-abfa-46a7-b432-3f428d5b05c0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:33:13.819372 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.819322    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409b00a9-abfa-46a7-b432-3f428d5b05c0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "409b00a9-abfa-46a7-b432-3f428d5b05c0" (UID: "409b00a9-abfa-46a7-b432-3f428d5b05c0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:33:13.819444 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.819421    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "409b00a9-abfa-46a7-b432-3f428d5b05c0" (UID: "409b00a9-abfa-46a7-b432-3f428d5b05c0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:33:13.821335 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.821311    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-api-access-n68bp" (OuterVolumeSpecName: "kube-api-access-n68bp") pod "409b00a9-abfa-46a7-b432-3f428d5b05c0" (UID: "409b00a9-abfa-46a7-b432-3f428d5b05c0"). InnerVolumeSpecName "kube-api-access-n68bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:33:13.821335 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.821320    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/409b00a9-abfa-46a7-b432-3f428d5b05c0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "409b00a9-abfa-46a7-b432-3f428d5b05c0" (UID: "409b00a9-abfa-46a7-b432-3f428d5b05c0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:33:13.919930 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.919889    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/409b00a9-abfa-46a7-b432-3f428d5b05c0-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:33:13.919930 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.919919    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/409b00a9-abfa-46a7-b432-3f428d5b05c0-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:33:13.919930 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.919930    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68bp\" (UniqueName: \"kubernetes.io/projected/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-api-access-n68bp\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:33:13.919930 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.919938    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:33:13.920200 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:13.919949    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/409b00a9-abfa-46a7-b432-3f428d5b05c0-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:33:14.521097 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.521066    2580 generic.go:356] "Generic (PLEG): container finished" podID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerID="6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f" exitCode=0
Sep 09 21:33:14.521271 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.521125    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"
Sep 09 21:33:14.521271 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.521149    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" event={"ID":"409b00a9-abfa-46a7-b432-3f428d5b05c0","Type":"ContainerDied","Data":"6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f"}
Sep 09 21:33:14.521271 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.521209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6" event={"ID":"409b00a9-abfa-46a7-b432-3f428d5b05c0","Type":"ContainerDied","Data":"eb593fc501179389a68efc6a8822de8a5ad453c6673c151ecb4043ac5e3efef0"}
Sep 09 21:33:14.521271 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.521262    2580 scope.go:117] "RemoveContainer" containerID="ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd"
Sep 09 21:33:14.530344 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.530324    2580 scope.go:117] "RemoveContainer" containerID="6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f"
Sep 09 21:33:14.537952 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.537936    2580 scope.go:117] "RemoveContainer" containerID="238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b"
Sep 09 21:33:14.542703 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.542682    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"]
Sep 09 21:33:14.547002 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.546980    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-v97w6"]
Sep 09 21:33:14.547204 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.547181    2580 scope.go:117] "RemoveContainer" containerID="ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd"
Sep 09 21:33:14.547482 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:33:14.547464    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd\": container with ID starting with ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd not found: ID does not exist" containerID="ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd"
Sep 09 21:33:14.547532 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.547488    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd"} err="failed to get container status \"ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd\": rpc error: code = NotFound desc = could not find container \"ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd\": container with ID starting with ad4b0d718441c4344cb31e54b1c10ff9e220e9406b4e711f192e440eda7fcadd not found: ID does not exist"
Sep 09 21:33:14.547532 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.547503    2580 scope.go:117] "RemoveContainer" containerID="6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f"
Sep 09 21:33:14.547744 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:33:14.547727    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f\": container with ID starting with 6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f not found: ID does not exist" containerID="6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f"
Sep 09 21:33:14.547818 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.547750    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f"} err="failed to get container status \"6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f\": rpc error: code = NotFound desc = could not find container \"6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f\": container with ID starting with 6057e66da4634cdcb9570a681ca80fc16d64ba4ddc1d8446f15bfd862f5bbf6f not found: ID does not exist"
Sep 09 21:33:14.547818 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.547784    2580 scope.go:117] "RemoveContainer" containerID="238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b"
Sep 09 21:33:14.548026 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:33:14.548009    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b\": container with ID starting with 238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b not found: ID does not exist" containerID="238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b"
Sep 09 21:33:14.548073 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.548032    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b"} err="failed to get container status \"238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b\": rpc error: code = NotFound desc = could not find container \"238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b\": container with ID starting with 238f3bbbd46253913735e56d9062953c7d1520dd65e24c3b454e4fd7d340790b not found: ID does not exist"
Sep 09 21:33:14.935983 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:33:14.935947    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" path="/var/lib/kubelet/pods/409b00a9-abfa-46a7-b432-3f428d5b05c0/volumes"
Sep 09 21:34:27.140480 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.140444    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"]
Sep 09 21:34:27.140906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.140796    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kube-rbac-proxy"
Sep 09 21:34:27.140906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.140808    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kube-rbac-proxy"
Sep 09 21:34:27.140906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.140818    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kserve-container"
Sep 09 21:34:27.140906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.140823    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kserve-container"
Sep 09 21:34:27.140906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.140836    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="storage-initializer"
Sep 09 21:34:27.140906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.140842    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="storage-initializer"
Sep 09 21:34:27.140906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.140899    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kserve-container"
Sep 09 21:34:27.140906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.140909    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="409b00a9-abfa-46a7-b432-3f428d5b05c0" containerName="kube-rbac-proxy"
Sep 09 21:34:27.144144 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.144124    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.146390 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.146371    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:34:27.146584 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.146566    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:34:27.146986 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.146965    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:34:27.147084 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.147067    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 09 21:34:27.147136 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.147070    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:34:27.147298 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.147282    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:34:27.147390 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.147371    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:34:27.155694 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.155670    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"]
Sep 09 21:34:27.223426 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.223389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.223600 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.223455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.223600 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.223497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.223600 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.223523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpcms\" (UniqueName: \"kubernetes.io/projected/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-api-access-bpcms\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.223717 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.223601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.324686 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.324649    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.324902 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.324702    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.324902 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.324745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.324902 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.324798    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.324902 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.324816    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpcms\" (UniqueName: \"kubernetes.io/projected/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-api-access-bpcms\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.325241 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.325217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.325532 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.325506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.325583 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.325506    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.327265 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.327245    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.334003 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.333983    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpcms\" (UniqueName: \"kubernetes.io/projected/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-api-access-bpcms\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.454893 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.454789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:27.734231 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.734157    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"]
Sep 09 21:34:27.736487 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:34:27.736460    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a84c7d8_c20e_4f9c_b7a2_91e8df1a8505.slice/crio-19c981548b8aba56af842d8a9a56e9fdff519be1a8eb67b3f5a78929602ae3ed WatchSource:0}: Error finding container 19c981548b8aba56af842d8a9a56e9fdff519be1a8eb67b3f5a78929602ae3ed: Status 404 returned error can't find the container with id 19c981548b8aba56af842d8a9a56e9fdff519be1a8eb67b3f5a78929602ae3ed
Sep 09 21:34:27.862217 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.862176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" event={"ID":"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505","Type":"ContainerStarted","Data":"04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1"}
Sep 09 21:34:27.862425 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:27.862257    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" event={"ID":"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505","Type":"ContainerStarted","Data":"19c981548b8aba56af842d8a9a56e9fdff519be1a8eb67b3f5a78929602ae3ed"}
Sep 09 21:34:31.883556 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:31.883515    2580 generic.go:356] "Generic (PLEG): container finished" podID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerID="04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1" exitCode=0
Sep 09 21:34:31.884067 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:31.883590    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" event={"ID":"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505","Type":"ContainerDied","Data":"04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1"}
Sep 09 21:34:32.890835 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:32.890802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" event={"ID":"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505","Type":"ContainerStarted","Data":"3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8"}
Sep 09 21:34:32.891224 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:32.890862    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" event={"ID":"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505","Type":"ContainerStarted","Data":"7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314"}
Sep 09 21:34:32.891224 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:32.891030    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:32.913341 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:32.913290    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" podStartSLOduration=2.141782933 podStartE2EDuration="5.91327613s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.771493197s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:34:32.911636119 +0000 UTC m=+2562.472663436" watchObservedRunningTime="2026-09-09 21:34:32.91327613 +0000 UTC m=+2562.474303447"
Sep 09 21:34:33.895501 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:33.895469    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:34:39.906896 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:34:39.906868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:35:09.911519 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:09.911483    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:35:17.241657 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:17.241575    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"]
Sep 09 21:35:17.242075 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:17.241818    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="kserve-container" containerID="cri-o://7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314" gracePeriod=30
Sep 09 21:35:17.242075 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:17.241857    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="kube-rbac-proxy" containerID="cri-o://3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8" gracePeriod=30
Sep 09 21:35:18.103321 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.103286    2580 generic.go:356] "Generic (PLEG): container finished" podID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerID="3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8" exitCode=2
Sep 09 21:35:18.103472 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.103363    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" event={"ID":"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505","Type":"ContainerDied","Data":"3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8"}
Sep 09 21:35:18.287370 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.287348    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:35:18.359413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.359383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-proxy-tls\") pod \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") "
Sep 09 21:35:18.359593 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.359452    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kserve-provision-location\") pod \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") "
Sep 09 21:35:18.359666 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.359615    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpcms\" (UniqueName: \"kubernetes.io/projected/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-api-access-bpcms\") pod \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") "
Sep 09 21:35:18.359666 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.359653    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-rbac-proxy-sar-config\") pod \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") "
Sep 09 21:35:18.359813 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.359687    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-cabundle-cert\") pod \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\" (UID: \"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505\") "
Sep 09 21:35:18.359892 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.359829    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" (UID: "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:35:18.360006 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.359990    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:35:18.360107 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.360077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" (UID: "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:18.360334 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.360117    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" (UID: "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:35:18.361727 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.361701    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" (UID: "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:35:18.361933 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.361746    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-api-access-bpcms" (OuterVolumeSpecName: "kube-api-access-bpcms") pod "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" (UID: "8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505"). InnerVolumeSpecName "kube-api-access-bpcms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:35:18.460924 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.460886    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpcms\" (UniqueName: \"kubernetes.io/projected/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-api-access-bpcms\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:35:18.460924 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.460915    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:35:18.460924 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.460927    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:35:18.460924 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:18.460936    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:35:19.110429 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.110399    2580 generic.go:356] "Generic (PLEG): container finished" podID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerID="7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314" exitCode=0
Sep 09 21:35:19.110639 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.110470    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"
Sep 09 21:35:19.110639 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.110490    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" event={"ID":"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505","Type":"ContainerDied","Data":"7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314"}
Sep 09 21:35:19.110639 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.110548    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2" event={"ID":"8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505","Type":"ContainerDied","Data":"19c981548b8aba56af842d8a9a56e9fdff519be1a8eb67b3f5a78929602ae3ed"}
Sep 09 21:35:19.110639 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.110573    2580 scope.go:117] "RemoveContainer" containerID="3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8"
Sep 09 21:35:19.119091 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.119069    2580 scope.go:117] "RemoveContainer" containerID="7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314"
Sep 09 21:35:19.130029 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.130004    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"]
Sep 09 21:35:19.131224 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.131207    2580 scope.go:117] "RemoveContainer" containerID="04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1"
Sep 09 21:35:19.135087 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.135061    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-wxzf2"]
Sep 09 21:35:19.140048 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.140029    2580 scope.go:117] "RemoveContainer" containerID="3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8"
Sep 09 21:35:19.140325 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:35:19.140306    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8\": container with ID starting with 3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8 not found: ID does not exist" containerID="3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8"
Sep 09 21:35:19.140370 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.140334    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8"} err="failed to get container status \"3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8\": rpc error: code = NotFound desc = could not find container \"3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8\": container with ID starting with 3413fd88204acf36ad7805e3c1bf4d7dac783832806f70a05e75a80ef3c0e4f8 not found: ID does not exist"
Sep 09 21:35:19.140370 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.140353    2580 scope.go:117] "RemoveContainer" containerID="7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314"
Sep 09 21:35:19.140618 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:35:19.140600    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314\": container with ID starting with 7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314 not found: ID does not exist" containerID="7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314"
Sep 09 21:35:19.140672 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.140623    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314"} err="failed to get container status \"7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314\": rpc error: code = NotFound desc = could not find container \"7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314\": container with ID starting with 7da55d2f8fb1970d3c4301ff5d85d62167cd1393bcc85781f824e9cf6297f314 not found: ID does not exist"
Sep 09 21:35:19.140672 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.140639    2580 scope.go:117] "RemoveContainer" containerID="04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1"
Sep 09 21:35:19.140931 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:35:19.140915    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1\": container with ID starting with 04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1 not found: ID does not exist" containerID="04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1"
Sep 09 21:35:19.140981 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:19.140937    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1"} err="failed to get container status \"04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1\": rpc error: code = NotFound desc = could not find container \"04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1\": container with ID starting with 04969f0a3d0ad30d6ff09621cbefe3c88563ad1fb1f3017a03394ff1c47760a1 not found: ID does not exist"
Sep 09 21:35:20.935800 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:20.935743    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" path="/var/lib/kubelet/pods/8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505/volumes"
Sep 09 21:35:57.669681 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.669642    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"]
Sep 09 21:35:57.670094 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.670028    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="kube-rbac-proxy"
Sep 09 21:35:57.670094 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.670041    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="kube-rbac-proxy"
Sep 09 21:35:57.670094 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.670053    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="kserve-container"
Sep 09 21:35:57.670094 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.670060    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="kserve-container"
Sep 09 21:35:57.670094 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.670067    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="storage-initializer"
Sep 09 21:35:57.670094 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.670075    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="storage-initializer"
Sep 09 21:35:57.670311 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.670144    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="kube-rbac-proxy"
Sep 09 21:35:57.670311 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.670156    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a84c7d8-c20e-4f9c-b7a2-91e8df1a8505" containerName="kserve-container"
Sep 09 21:35:57.674490 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.674472    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.676891 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.676868    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 09 21:35:57.677021 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.676916    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 09 21:35:57.677021 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.676969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:35:57.677620 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.677603    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:35:57.677693 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.677606    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 09 21:35:57.677693 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.677634    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:35:57.679714 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.679697    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:35:57.688597 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.688576    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"]
Sep 09 21:35:57.804418 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.804389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ef636c-3828-4e6d-a5b5-e7dad38f6322-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.804603 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.804444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.804603 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.804498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.804603 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.804580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.804737 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.804618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szhh2\" (UniqueName: \"kubernetes.io/projected/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-api-access-szhh2\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.905587 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.905553    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szhh2\" (UniqueName: \"kubernetes.io/projected/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-api-access-szhh2\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.905587 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.905601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ef636c-3828-4e6d-a5b5-e7dad38f6322-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.905869 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.905728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.905869 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.905800    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.905953 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.905874    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.906213 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.906192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.906431 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.906415    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.906477 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.906449    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.908128 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.908108    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ef636c-3828-4e6d-a5b5-e7dad38f6322-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.916699 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.916674    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szhh2\" (UniqueName: \"kubernetes.io/projected/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-api-access-szhh2\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:57.984956 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:57.984875    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:35:58.262669 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:58.262640    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"]
Sep 09 21:35:58.264958 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:35:58.264932    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ef636c_3828_4e6d_a5b5_e7dad38f6322.slice/crio-026c954ef764fb1d0eafc902df86ec85e8930b45f9bb652abd5e665b2eb780b1 WatchSource:0}: Error finding container 026c954ef764fb1d0eafc902df86ec85e8930b45f9bb652abd5e665b2eb780b1: Status 404 returned error can't find the container with id 026c954ef764fb1d0eafc902df86ec85e8930b45f9bb652abd5e665b2eb780b1
Sep 09 21:35:58.290199 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:58.290176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" event={"ID":"31ef636c-3828-4e6d-a5b5-e7dad38f6322","Type":"ContainerStarted","Data":"026c954ef764fb1d0eafc902df86ec85e8930b45f9bb652abd5e665b2eb780b1"}
Sep 09 21:35:59.296956 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:35:59.296922    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" event={"ID":"31ef636c-3828-4e6d-a5b5-e7dad38f6322","Type":"ContainerStarted","Data":"58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38"}
Sep 09 21:36:02.314024 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:02.313989    2580 generic.go:356] "Generic (PLEG): container finished" podID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerID="58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38" exitCode=0
Sep 09 21:36:02.314378 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:02.314066    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" event={"ID":"31ef636c-3828-4e6d-a5b5-e7dad38f6322","Type":"ContainerDied","Data":"58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38"}
Sep 09 21:36:03.321740 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:03.321703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" event={"ID":"31ef636c-3828-4e6d-a5b5-e7dad38f6322","Type":"ContainerStarted","Data":"0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6"}
Sep 09 21:36:03.322236 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:03.321793    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" event={"ID":"31ef636c-3828-4e6d-a5b5-e7dad38f6322","Type":"ContainerStarted","Data":"1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02"}
Sep 09 21:36:03.322236 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:03.322014    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:36:03.345611 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:03.345557    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" podStartSLOduration=2.426222293 podStartE2EDuration="6.345541597s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.919319304s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:36:03.343611708 +0000 UTC m=+2652.904639026" watchObservedRunningTime="2026-09-09 21:36:03.345541597 +0000 UTC m=+2652.906568913"
Sep 09 21:36:04.326285 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:04.326254    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:36:10.336075 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:10.336045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:36:40.340857 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:40.340819    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:36:47.692473 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:47.692439    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"]
Sep 09 21:36:47.692914 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:47.692685    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="kserve-container" containerID="cri-o://1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02" gracePeriod=30
Sep 09 21:36:47.692914 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:47.692721    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="kube-rbac-proxy" containerID="cri-o://0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6" gracePeriod=30
Sep 09 21:36:48.545420 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:48.545385    2580 generic.go:356] "Generic (PLEG): container finished" podID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerID="0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6" exitCode=2
Sep 09 21:36:48.545606 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:48.545462    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" event={"ID":"31ef636c-3828-4e6d-a5b5-e7dad38f6322","Type":"ContainerDied","Data":"0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6"}
Sep 09 21:36:48.872007 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:48.871982    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:36:49.029501 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.029464    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ef636c-3828-4e6d-a5b5-e7dad38f6322-proxy-tls\") pod \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") "
Sep 09 21:36:49.029703 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.029536    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szhh2\" (UniqueName: \"kubernetes.io/projected/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-api-access-szhh2\") pod \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") "
Sep 09 21:36:49.029703 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.029569    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-rbac-proxy-sar-config\") pod \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") "
Sep 09 21:36:49.029703 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.029604    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kserve-provision-location\") pod \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") "
Sep 09 21:36:49.029703 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.029625    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-cabundle-cert\") pod \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\" (UID: \"31ef636c-3828-4e6d-a5b5-e7dad38f6322\") "
Sep 09 21:36:49.030031 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.030003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31ef636c-3828-4e6d-a5b5-e7dad38f6322" (UID: "31ef636c-3828-4e6d-a5b5-e7dad38f6322"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:36:49.030031 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.030014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "31ef636c-3828-4e6d-a5b5-e7dad38f6322" (UID: "31ef636c-3828-4e6d-a5b5-e7dad38f6322"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:36:49.030176 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.030003    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "31ef636c-3828-4e6d-a5b5-e7dad38f6322" (UID: "31ef636c-3828-4e6d-a5b5-e7dad38f6322"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:36:49.032419 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.032393    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ef636c-3828-4e6d-a5b5-e7dad38f6322-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "31ef636c-3828-4e6d-a5b5-e7dad38f6322" (UID: "31ef636c-3828-4e6d-a5b5-e7dad38f6322"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:36:49.032419 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.032396    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-api-access-szhh2" (OuterVolumeSpecName: "kube-api-access-szhh2") pod "31ef636c-3828-4e6d-a5b5-e7dad38f6322" (UID: "31ef636c-3828-4e6d-a5b5-e7dad38f6322"). InnerVolumeSpecName "kube-api-access-szhh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:36:49.131010 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.130981    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szhh2\" (UniqueName: \"kubernetes.io/projected/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-api-access-szhh2\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:36:49.131010 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.131013    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:36:49.131189 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.131024    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31ef636c-3828-4e6d-a5b5-e7dad38f6322-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:36:49.131189 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.131036    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/31ef636c-3828-4e6d-a5b5-e7dad38f6322-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:36:49.131189 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.131044    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/31ef636c-3828-4e6d-a5b5-e7dad38f6322-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:36:49.552282 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.552192    2580 generic.go:356] "Generic (PLEG): container finished" podID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerID="1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02" exitCode=0
Sep 09 21:36:49.552282 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.552252    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"
Sep 09 21:36:49.552462 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.552279    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" event={"ID":"31ef636c-3828-4e6d-a5b5-e7dad38f6322","Type":"ContainerDied","Data":"1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02"}
Sep 09 21:36:49.552462 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.552344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm" event={"ID":"31ef636c-3828-4e6d-a5b5-e7dad38f6322","Type":"ContainerDied","Data":"026c954ef764fb1d0eafc902df86ec85e8930b45f9bb652abd5e665b2eb780b1"}
Sep 09 21:36:49.552462 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.552370    2580 scope.go:117] "RemoveContainer" containerID="0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6"
Sep 09 21:36:49.568422 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.568390    2580 scope.go:117] "RemoveContainer" containerID="1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02"
Sep 09 21:36:49.577054 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.577034    2580 scope.go:117] "RemoveContainer" containerID="58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38"
Sep 09 21:36:49.578459 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.578440    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"]
Sep 09 21:36:49.582209 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.582186    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2cqnm"]
Sep 09 21:36:49.585293 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.585275    2580 scope.go:117] "RemoveContainer" containerID="0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6"
Sep 09 21:36:49.585573 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:36:49.585554    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6\": container with ID starting with 0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6 not found: ID does not exist" containerID="0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6"
Sep 09 21:36:49.585627 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.585582    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6"} err="failed to get container status \"0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6\": rpc error: code = NotFound desc = could not find container \"0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6\": container with ID starting with 0d157b7f897332b4e1ecbd2600baa5f8ad17beba888d1020f2b09d31229608a6 not found: ID does not exist"
Sep 09 21:36:49.585627 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.585601    2580 scope.go:117] "RemoveContainer" containerID="1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02"
Sep 09 21:36:49.585991 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:36:49.585972    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02\": container with ID starting with 1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02 not found: ID does not exist" containerID="1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02"
Sep 09 21:36:49.586060 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.585996    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02"} err="failed to get container status \"1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02\": rpc error: code = NotFound desc = could not find container \"1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02\": container with ID starting with 1b27c6d4218bf200f65069afb0a8a3e7e83f7214440b1e04f4bd8dd7daac1a02 not found: ID does not exist"
Sep 09 21:36:49.586060 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.586012    2580 scope.go:117] "RemoveContainer" containerID="58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38"
Sep 09 21:36:49.586263 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:36:49.586243    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38\": container with ID starting with 58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38 not found: ID does not exist" containerID="58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38"
Sep 09 21:36:49.586324 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:49.586282    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38"} err="failed to get container status \"58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38\": rpc error: code = NotFound desc = could not find container \"58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38\": container with ID starting with 58af52531d55a67f4766d6ca1233071f2cb0a23d5444a828f560a6c8c9654d38 not found: ID does not exist"
Sep 09 21:36:50.936019 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:36:50.935992    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" path="/var/lib/kubelet/pods/31ef636c-3828-4e6d-a5b5-e7dad38f6322/volumes"
Sep 09 21:37:05.136397 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:05.136290    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:37:05.140325 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:05.140315    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:37:42.066849 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.066812    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"]
Sep 09 21:37:42.067310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.067186    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="kserve-container"
Sep 09 21:37:42.067310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.067197    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="kserve-container"
Sep 09 21:37:42.067310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.067206    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="storage-initializer"
Sep 09 21:37:42.067310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.067212    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="storage-initializer"
Sep 09 21:37:42.067310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.067218    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="kube-rbac-proxy"
Sep 09 21:37:42.067310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.067223    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="kube-rbac-proxy"
Sep 09 21:37:42.067310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.067280    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="kserve-container"
Sep 09 21:37:42.067310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.067291    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ef636c-3828-4e6d-a5b5-e7dad38f6322" containerName="kube-rbac-proxy"
Sep 09 21:37:42.070380 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.070364    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.072503 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.072485    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 21:37:42.072606 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.072589    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:37:42.072645 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.072629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:37:42.072807 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.072795    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:37:42.072870 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.072855    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 09 21:37:42.073246 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.073224    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 09 21:37:42.081250 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.081230    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"]
Sep 09 21:37:42.147302 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.147260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25mms\" (UniqueName: \"kubernetes.io/projected/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-api-access-25mms\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.147475 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.147320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d0dfb3-179e-4462-b656-d52c1121d103-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.147475 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.147367    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d0dfb3-179e-4462-b656-d52c1121d103-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.147475 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.147421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.248633 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.248594    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d0dfb3-179e-4462-b656-d52c1121d103-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.248811 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.248653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d0dfb3-179e-4462-b656-d52c1121d103-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.248811 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.248678    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.248811 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.248718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25mms\" (UniqueName: \"kubernetes.io/projected/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-api-access-25mms\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.249101 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.249075    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d0dfb3-179e-4462-b656-d52c1121d103-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.249510 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.249488    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.251264 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.251237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d0dfb3-179e-4462-b656-d52c1121d103-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.256263 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.256244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25mms\" (UniqueName: \"kubernetes.io/projected/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-api-access-25mms\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.381311 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.381274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:42.654302 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.654244    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"]
Sep 09 21:37:42.656554 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:37:42.656525    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9d0dfb3_179e_4462_b656_d52c1121d103.slice/crio-bd2e72bd959482dc4ba02a1c27e27b4dc0db603feec8612ab283ebe1a5337e34 WatchSource:0}: Error finding container bd2e72bd959482dc4ba02a1c27e27b4dc0db603feec8612ab283ebe1a5337e34: Status 404 returned error can't find the container with id bd2e72bd959482dc4ba02a1c27e27b4dc0db603feec8612ab283ebe1a5337e34
Sep 09 21:37:42.792386 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.792342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd" event={"ID":"a9d0dfb3-179e-4462-b656-d52c1121d103","Type":"ContainerStarted","Data":"0d8b264451729148e387c4216f93628924653aba5278146cc6b0b72ab713b622"}
Sep 09 21:37:42.792583 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:42.792420    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd" event={"ID":"a9d0dfb3-179e-4462-b656-d52c1121d103","Type":"ContainerStarted","Data":"bd2e72bd959482dc4ba02a1c27e27b4dc0db603feec8612ab283ebe1a5337e34"}
Sep 09 21:37:46.815133 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:46.815054    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_a9d0dfb3-179e-4462-b656-d52c1121d103/storage-initializer/0.log"
Sep 09 21:37:46.815133 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:46.815091    2580 generic.go:356] "Generic (PLEG): container finished" podID="a9d0dfb3-179e-4462-b656-d52c1121d103" containerID="0d8b264451729148e387c4216f93628924653aba5278146cc6b0b72ab713b622" exitCode=1
Sep 09 21:37:46.815516 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:46.815142    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd" event={"ID":"a9d0dfb3-179e-4462-b656-d52c1121d103","Type":"ContainerDied","Data":"0d8b264451729148e387c4216f93628924653aba5278146cc6b0b72ab713b622"}
Sep 09 21:37:47.822236 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:47.822206    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_a9d0dfb3-179e-4462-b656-d52c1121d103/storage-initializer/0.log"
Sep 09 21:37:47.822661 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:47.822278    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd" event={"ID":"a9d0dfb3-179e-4462-b656-d52c1121d103","Type":"ContainerStarted","Data":"ddd4c3d457874e3b08cb115931a63b3cf0b9d841a00e5fe7bdac3c6b5770a7a9"}
Sep 09 21:37:50.838995 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:50.838962    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_a9d0dfb3-179e-4462-b656-d52c1121d103/storage-initializer/1.log"
Sep 09 21:37:50.839397 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:50.839382    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_a9d0dfb3-179e-4462-b656-d52c1121d103/storage-initializer/0.log"
Sep 09 21:37:50.839454 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:50.839422    2580 generic.go:356] "Generic (PLEG): container finished" podID="a9d0dfb3-179e-4462-b656-d52c1121d103" containerID="ddd4c3d457874e3b08cb115931a63b3cf0b9d841a00e5fe7bdac3c6b5770a7a9" exitCode=1
Sep 09 21:37:50.839525 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:50.839500    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd" event={"ID":"a9d0dfb3-179e-4462-b656-d52c1121d103","Type":"ContainerDied","Data":"ddd4c3d457874e3b08cb115931a63b3cf0b9d841a00e5fe7bdac3c6b5770a7a9"}
Sep 09 21:37:50.839602 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:50.839590    2580 scope.go:117] "RemoveContainer" containerID="0d8b264451729148e387c4216f93628924653aba5278146cc6b0b72ab713b622"
Sep 09 21:37:50.839988 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:50.839963    2580 scope.go:117] "RemoveContainer" containerID="0d8b264451729148e387c4216f93628924653aba5278146cc6b0b72ab713b622"
Sep 09 21:37:50.850793 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:37:50.850752    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_kserve-ci-e2e-test(a9d0dfb3-179e-4462-b656-d52c1121d103)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd" podUID="a9d0dfb3-179e-4462-b656-d52c1121d103"
Sep 09 21:37:50.850939 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:37:50.850917    2580 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_kserve-ci-e2e-test_a9d0dfb3-179e-4462-b656-d52c1121d103_0 in pod sandbox bd2e72bd959482dc4ba02a1c27e27b4dc0db603feec8612ab283ebe1a5337e34 from index: no such id: '0d8b264451729148e387c4216f93628924653aba5278146cc6b0b72ab713b622'" containerID="0d8b264451729148e387c4216f93628924653aba5278146cc6b0b72ab713b622"
Sep 09 21:37:50.850988 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:50.850949    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8b264451729148e387c4216f93628924653aba5278146cc6b0b72ab713b622"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_kserve-ci-e2e-test_a9d0dfb3-179e-4462-b656-d52c1121d103_0 in pod sandbox bd2e72bd959482dc4ba02a1c27e27b4dc0db603feec8612ab283ebe1a5337e34 from index: no such id: '0d8b264451729148e387c4216f93628924653aba5278146cc6b0b72ab713b622'"
Sep 09 21:37:51.845419 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:51.845395    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_a9d0dfb3-179e-4462-b656-d52c1121d103/storage-initializer/1.log"
Sep 09 21:37:52.052487 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.052446    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"]
Sep 09 21:37:52.185698 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.185677    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_a9d0dfb3-179e-4462-b656-d52c1121d103/storage-initializer/1.log"
Sep 09 21:37:52.189197 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.189182    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:52.339966 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.339921    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d0dfb3-179e-4462-b656-d52c1121d103-kserve-provision-location\") pod \"a9d0dfb3-179e-4462-b656-d52c1121d103\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") "
Sep 09 21:37:52.339966 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.339968    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d0dfb3-179e-4462-b656-d52c1121d103-proxy-tls\") pod \"a9d0dfb3-179e-4462-b656-d52c1121d103\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") "
Sep 09 21:37:52.340231 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.340041    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-rbac-proxy-sar-config\") pod \"a9d0dfb3-179e-4462-b656-d52c1121d103\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") "
Sep 09 21:37:52.340231 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.340098    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25mms\" (UniqueName: \"kubernetes.io/projected/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-api-access-25mms\") pod \"a9d0dfb3-179e-4462-b656-d52c1121d103\" (UID: \"a9d0dfb3-179e-4462-b656-d52c1121d103\") "
Sep 09 21:37:52.340317 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.340253    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9d0dfb3-179e-4462-b656-d52c1121d103-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a9d0dfb3-179e-4462-b656-d52c1121d103" (UID: "a9d0dfb3-179e-4462-b656-d52c1121d103"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:37:52.340369 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.340359    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a9d0dfb3-179e-4462-b656-d52c1121d103-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:37:52.340518 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.340490    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a9d0dfb3-179e-4462-b656-d52c1121d103" (UID: "a9d0dfb3-179e-4462-b656-d52c1121d103"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:37:52.342304 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.342277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9d0dfb3-179e-4462-b656-d52c1121d103-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a9d0dfb3-179e-4462-b656-d52c1121d103" (UID: "a9d0dfb3-179e-4462-b656-d52c1121d103"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:37:52.342406 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.342387    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-api-access-25mms" (OuterVolumeSpecName: "kube-api-access-25mms") pod "a9d0dfb3-179e-4462-b656-d52c1121d103" (UID: "a9d0dfb3-179e-4462-b656-d52c1121d103"). InnerVolumeSpecName "kube-api-access-25mms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:37:52.440939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.440858    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25mms\" (UniqueName: \"kubernetes.io/projected/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-api-access-25mms\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:37:52.440939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.440885    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a9d0dfb3-179e-4462-b656-d52c1121d103-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:37:52.440939 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.440895    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a9d0dfb3-179e-4462-b656-d52c1121d103-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:37:52.852159 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.852135    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd_a9d0dfb3-179e-4462-b656-d52c1121d103/storage-initializer/1.log"
Sep 09 21:37:52.852623 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.852243    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"
Sep 09 21:37:52.852623 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.852264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd" event={"ID":"a9d0dfb3-179e-4462-b656-d52c1121d103","Type":"ContainerDied","Data":"bd2e72bd959482dc4ba02a1c27e27b4dc0db603feec8612ab283ebe1a5337e34"}
Sep 09 21:37:52.852623 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.852326    2580 scope.go:117] "RemoveContainer" containerID="ddd4c3d457874e3b08cb115931a63b3cf0b9d841a00e5fe7bdac3c6b5770a7a9"
Sep 09 21:37:52.887837 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.887804    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"]
Sep 09 21:37:52.891254 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.891229    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-69d586f7f9-zkxsd"]
Sep 09 21:37:52.938984 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:52.936828    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9d0dfb3-179e-4462-b656-d52c1121d103" path="/var/lib/kubelet/pods/a9d0dfb3-179e-4462-b656-d52c1121d103/volumes"
Sep 09 21:37:53.180694 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.180622    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"]
Sep 09 21:37:53.180998 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.180985    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d0dfb3-179e-4462-b656-d52c1121d103" containerName="storage-initializer"
Sep 09 21:37:53.181057 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.180999    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d0dfb3-179e-4462-b656-d52c1121d103" containerName="storage-initializer"
Sep 09 21:37:53.181057 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.181010    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9d0dfb3-179e-4462-b656-d52c1121d103" containerName="storage-initializer"
Sep 09 21:37:53.181057 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.181016    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9d0dfb3-179e-4462-b656-d52c1121d103" containerName="storage-initializer"
Sep 09 21:37:53.181149 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.181071    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d0dfb3-179e-4462-b656-d52c1121d103" containerName="storage-initializer"
Sep 09 21:37:53.181149 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.181080    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9d0dfb3-179e-4462-b656-d52c1121d103" containerName="storage-initializer"
Sep 09 21:37:53.185660 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.185643    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.188026 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.187994    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 09 21:37:53.188157 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.188045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 21:37:53.188157 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.188057    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 09 21:37:53.188157 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.188072    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:37:53.188319 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.188303    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:37:53.188390 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.188373    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:37:53.188446 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.188391    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:37:53.196480 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.196458    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"]
Sep 09 21:37:53.348184 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.348141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66164e11-fdf5-446c-adee-c078fa8e6f29-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.348184 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.348187    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66164e11-fdf5-446c-adee-c078fa8e6f29-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.348411 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.348204    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.348411 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.348293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.348411 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.348375    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49kpk\" (UniqueName: \"kubernetes.io/projected/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-api-access-49kpk\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.449675 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.449573    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.449675 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.449645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49kpk\" (UniqueName: \"kubernetes.io/projected/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-api-access-49kpk\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.449964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.449701    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66164e11-fdf5-446c-adee-c078fa8e6f29-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.449964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.449747    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66164e11-fdf5-446c-adee-c078fa8e6f29-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.449964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.449804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.450169 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.450148    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66164e11-fdf5-446c-adee-c078fa8e6f29-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.450591 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.450502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.450591 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.450504    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.452394 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.452374    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66164e11-fdf5-446c-adee-c078fa8e6f29-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.458159 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.458138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49kpk\" (UniqueName: \"kubernetes.io/projected/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-api-access-49kpk\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.496853 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.496817    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:53.772848 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.771503    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"]
Sep 09 21:37:53.775043 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:37:53.775015    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66164e11_fdf5_446c_adee_c078fa8e6f29.slice/crio-5a487f334baff0b31474263c584cfaff6f265a72599bff4fcb13ba4cc0841879 WatchSource:0}: Error finding container 5a487f334baff0b31474263c584cfaff6f265a72599bff4fcb13ba4cc0841879: Status 404 returned error can't find the container with id 5a487f334baff0b31474263c584cfaff6f265a72599bff4fcb13ba4cc0841879
Sep 09 21:37:53.859838 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.859798    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" event={"ID":"66164e11-fdf5-446c-adee-c078fa8e6f29","Type":"ContainerStarted","Data":"37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47"}
Sep 09 21:37:53.860220 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:53.859874    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" event={"ID":"66164e11-fdf5-446c-adee-c078fa8e6f29","Type":"ContainerStarted","Data":"5a487f334baff0b31474263c584cfaff6f265a72599bff4fcb13ba4cc0841879"}
Sep 09 21:37:55.873105 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:55.873074    2580 generic.go:356] "Generic (PLEG): container finished" podID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerID="37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47" exitCode=0
Sep 09 21:37:55.873507 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:55.873148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" event={"ID":"66164e11-fdf5-446c-adee-c078fa8e6f29","Type":"ContainerDied","Data":"37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47"}
Sep 09 21:37:56.880912 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:56.880879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" event={"ID":"66164e11-fdf5-446c-adee-c078fa8e6f29","Type":"ContainerStarted","Data":"dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969"}
Sep 09 21:37:56.881283 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:56.880942    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" event={"ID":"66164e11-fdf5-446c-adee-c078fa8e6f29","Type":"ContainerStarted","Data":"3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c"}
Sep 09 21:37:56.881283 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:56.881122    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:56.902122 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:56.902076    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podStartSLOduration=2.85565581 podStartE2EDuration="3.90206087s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.04640506s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:37:56.901208111 +0000 UTC m=+2766.462235428" watchObservedRunningTime="2026-09-09 21:37:56.90206087 +0000 UTC m=+2766.463088191"
Sep 09 21:37:57.885631 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:57.885599    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:37:57.887021 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:57.886995    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:37:58.896266 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:37:58.896225    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:38:03.901941 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:38:03.901915    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:38:03.902434 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:38:03.902400    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:38:13.903157 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:38:13.903064    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:38:23.902359 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:38:23.902322    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:38:33.902460 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:38:33.902417    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:38:43.902549 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:38:43.902509    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:38:53.902639 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:38:53.902601    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:39:03.902902 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:03.902857    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:39:13.903669 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:13.903636    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:39:23.207520 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:23.207481    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"]
Sep 09 21:39:23.208050 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:23.207734    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" containerID="cri-o://3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c" gracePeriod=30
Sep 09 21:39:23.208050 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:23.207833    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kube-rbac-proxy" containerID="cri-o://dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969" gracePeriod=30
Sep 09 21:39:23.897305 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:23.897262    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.64:8643/healthz\": dial tcp 10.132.0.64:8643: connect: connection refused"
Sep 09 21:39:23.902495 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:23.902472    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.64:8080: connect: connection refused"
Sep 09 21:39:24.301391 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:24.301312    2580 generic.go:356] "Generic (PLEG): container finished" podID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerID="dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969" exitCode=2
Sep 09 21:39:24.301773 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:24.301392    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" event={"ID":"66164e11-fdf5-446c-adee-c078fa8e6f29","Type":"ContainerDied","Data":"dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969"}
Sep 09 21:39:28.056168 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.056143    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:39:28.135958 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.135879    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-cabundle-cert\") pod \"66164e11-fdf5-446c-adee-c078fa8e6f29\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") "
Sep 09 21:39:28.136118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.135970    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-rbac-proxy-sar-config\") pod \"66164e11-fdf5-446c-adee-c078fa8e6f29\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") "
Sep 09 21:39:28.136118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.135999    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66164e11-fdf5-446c-adee-c078fa8e6f29-kserve-provision-location\") pod \"66164e11-fdf5-446c-adee-c078fa8e6f29\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") "
Sep 09 21:39:28.136118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.136022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66164e11-fdf5-446c-adee-c078fa8e6f29-proxy-tls\") pod \"66164e11-fdf5-446c-adee-c078fa8e6f29\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") "
Sep 09 21:39:28.136118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.136043    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49kpk\" (UniqueName: \"kubernetes.io/projected/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-api-access-49kpk\") pod \"66164e11-fdf5-446c-adee-c078fa8e6f29\" (UID: \"66164e11-fdf5-446c-adee-c078fa8e6f29\") "
Sep 09 21:39:28.136327 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.136287    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66164e11-fdf5-446c-adee-c078fa8e6f29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66164e11-fdf5-446c-adee-c078fa8e6f29" (UID: "66164e11-fdf5-446c-adee-c078fa8e6f29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:39:28.136375 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.136342    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66164e11-fdf5-446c-adee-c078fa8e6f29" (UID: "66164e11-fdf5-446c-adee-c078fa8e6f29"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:39:28.136375 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.136360    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66164e11-fdf5-446c-adee-c078fa8e6f29" (UID: "66164e11-fdf5-446c-adee-c078fa8e6f29"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:39:28.138212 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.138177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-api-access-49kpk" (OuterVolumeSpecName: "kube-api-access-49kpk") pod "66164e11-fdf5-446c-adee-c078fa8e6f29" (UID: "66164e11-fdf5-446c-adee-c078fa8e6f29"). InnerVolumeSpecName "kube-api-access-49kpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:39:28.138212 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.138180    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66164e11-fdf5-446c-adee-c078fa8e6f29-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66164e11-fdf5-446c-adee-c078fa8e6f29" (UID: "66164e11-fdf5-446c-adee-c078fa8e6f29"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:39:28.236716 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.236685    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:39:28.236716 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.236711    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:39:28.236716 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.236721    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66164e11-fdf5-446c-adee-c078fa8e6f29-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:39:28.236964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.236730    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66164e11-fdf5-446c-adee-c078fa8e6f29-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:39:28.236964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.236739    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49kpk\" (UniqueName: \"kubernetes.io/projected/66164e11-fdf5-446c-adee-c078fa8e6f29-kube-api-access-49kpk\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:39:28.322565 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.322534    2580 generic.go:356] "Generic (PLEG): container finished" podID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerID="3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c" exitCode=0
Sep 09 21:39:28.322733 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.322588    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"
Sep 09 21:39:28.322733 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.322622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" event={"ID":"66164e11-fdf5-446c-adee-c078fa8e6f29","Type":"ContainerDied","Data":"3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c"}
Sep 09 21:39:28.322733 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.322681    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6" event={"ID":"66164e11-fdf5-446c-adee-c078fa8e6f29","Type":"ContainerDied","Data":"5a487f334baff0b31474263c584cfaff6f265a72599bff4fcb13ba4cc0841879"}
Sep 09 21:39:28.322733 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.322710    2580 scope.go:117] "RemoveContainer" containerID="dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969"
Sep 09 21:39:28.331540 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.331524    2580 scope.go:117] "RemoveContainer" containerID="3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c"
Sep 09 21:39:28.339140 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.339123    2580 scope.go:117] "RemoveContainer" containerID="37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47"
Sep 09 21:39:28.346026 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.346004    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"]
Sep 09 21:39:28.346996 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.346975    2580 scope.go:117] "RemoveContainer" containerID="dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969"
Sep 09 21:39:28.347242 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:39:28.347226    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969\": container with ID starting with dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969 not found: ID does not exist" containerID="dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969"
Sep 09 21:39:28.347293 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.347250    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969"} err="failed to get container status \"dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969\": rpc error: code = NotFound desc = could not find container \"dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969\": container with ID starting with dd3b112455fb14368a7341f4a1e67cc72058e0d0a2d51d91e976d906c16dc969 not found: ID does not exist"
Sep 09 21:39:28.347293 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.347265    2580 scope.go:117] "RemoveContainer" containerID="3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c"
Sep 09 21:39:28.347559 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:39:28.347542    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c\": container with ID starting with 3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c not found: ID does not exist" containerID="3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c"
Sep 09 21:39:28.347615 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.347565    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c"} err="failed to get container status \"3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c\": rpc error: code = NotFound desc = could not find container \"3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c\": container with ID starting with 3eab941b27d28c17fb5c035d7885c52e619dd8a299b6b29578142afc77172b7c not found: ID does not exist"
Sep 09 21:39:28.347615 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.347580    2580 scope.go:117] "RemoveContainer" containerID="37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47"
Sep 09 21:39:28.347868 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:39:28.347837    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47\": container with ID starting with 37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47 not found: ID does not exist" containerID="37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47"
Sep 09 21:39:28.347954 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.347871    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47"} err="failed to get container status \"37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47\": rpc error: code = NotFound desc = could not find container \"37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47\": container with ID starting with 37ec4bc30a667e4c4f9a88ce18f5b4064af32db9969ff8efe3683c585f1c6d47 not found: ID does not exist"
Sep 09 21:39:28.350587 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.350567    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-77678d899f-pqjx6"]
Sep 09 21:39:28.935914 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:28.935878    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" path="/var/lib/kubelet/pods/66164e11-fdf5-446c-adee-c078fa8e6f29/volumes"
Sep 09 21:39:35.441161 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.441124    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"]
Sep 09 21:39:35.441596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.441487    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container"
Sep 09 21:39:35.441596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.441498    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container"
Sep 09 21:39:35.441596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.441518    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="storage-initializer"
Sep 09 21:39:35.441596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.441523    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="storage-initializer"
Sep 09 21:39:35.441596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.441532    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kube-rbac-proxy"
Sep 09 21:39:35.441596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.441538    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kube-rbac-proxy"
Sep 09 21:39:35.441596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.441588    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kube-rbac-proxy"
Sep 09 21:39:35.441596 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.441597    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="66164e11-fdf5-446c-adee-c078fa8e6f29" containerName="kserve-container"
Sep 09 21:39:35.444486 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.444470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.447577 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.447550    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 09 21:39:35.447831 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.447710    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 09 21:39:35.447831 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.447795    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 09 21:39:35.447942 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.447879    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 09 21:39:35.448070 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.448055    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-xmxdq\""
Sep 09 21:39:35.448118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.448055    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 09 21:39:35.448118 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.448059    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 09 21:39:35.457588 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.457565    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"]
Sep 09 21:39:35.496083 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.496051    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8x9s\" (UniqueName: \"kubernetes.io/projected/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-api-access-w8x9s\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.496248 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.496100    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.496248 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.496138    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34c76101-e4a6-4171-bc46-e3d94641cca6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.496248 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.496185    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34c76101-e4a6-4171-bc46-e3d94641cca6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.496385 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.496249    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.596662 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.596625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34c76101-e4a6-4171-bc46-e3d94641cca6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.596662 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.596669    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.596958 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.596693    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8x9s\" (UniqueName: \"kubernetes.io/projected/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-api-access-w8x9s\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.596958 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.596723    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.596958 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.596750    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34c76101-e4a6-4171-bc46-e3d94641cca6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.597237 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.597215    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34c76101-e4a6-4171-bc46-e3d94641cca6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.597444 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.597423    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.597528 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.597430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.599310 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.599286    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34c76101-e4a6-4171-bc46-e3d94641cca6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.609153 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.609125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8x9s\" (UniqueName: \"kubernetes.io/projected/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-api-access-w8x9s\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:35.755140 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:35.755058    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:36.028903 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:36.028833    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"]
Sep 09 21:39:36.031194 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:39:36.031163    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34c76101_e4a6_4171_bc46_e3d94641cca6.slice/crio-89cc8fcdafdd8f301ef3fda49fc31973068cf14d8deac3d5d365fd908823d7da WatchSource:0}: Error finding container 89cc8fcdafdd8f301ef3fda49fc31973068cf14d8deac3d5d365fd908823d7da: Status 404 returned error can't find the container with id 89cc8fcdafdd8f301ef3fda49fc31973068cf14d8deac3d5d365fd908823d7da
Sep 09 21:39:36.363328 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:36.363286    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" event={"ID":"34c76101-e4a6-4171-bc46-e3d94641cca6","Type":"ContainerStarted","Data":"451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2"}
Sep 09 21:39:36.363474 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:36.363369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" event={"ID":"34c76101-e4a6-4171-bc46-e3d94641cca6","Type":"ContainerStarted","Data":"89cc8fcdafdd8f301ef3fda49fc31973068cf14d8deac3d5d365fd908823d7da"}
Sep 09 21:39:37.370474 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:37.370439    2580 generic.go:356] "Generic (PLEG): container finished" podID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerID="451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2" exitCode=0
Sep 09 21:39:37.370899 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:37.370527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" event={"ID":"34c76101-e4a6-4171-bc46-e3d94641cca6","Type":"ContainerDied","Data":"451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2"}
Sep 09 21:39:38.378260 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:38.378214    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" event={"ID":"34c76101-e4a6-4171-bc46-e3d94641cca6","Type":"ContainerStarted","Data":"4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7"}
Sep 09 21:39:38.378663 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:38.378295    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:38.378663 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:38.378324    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" event={"ID":"34c76101-e4a6-4171-bc46-e3d94641cca6","Type":"ContainerStarted","Data":"1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a"}
Sep 09 21:39:38.378663 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:38.378357    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:38.379557 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:38.379519    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 09 21:39:38.397827 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:38.397742    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podStartSLOduration=2.36950562 podStartE2EDuration="3.397724056s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.028218436s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:39:38.397114194 +0000 UTC m=+2867.958141535" watchObservedRunningTime="2026-09-09 21:39:38.397724056 +0000 UTC m=+2867.958751374"
Sep 09 21:39:39.383613 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:39.383561    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 09 21:39:44.387964 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:44.387935    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:39:44.388411 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:44.388385    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 09 21:39:54.389055 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:39:54.389011    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 09 21:40:04.388619 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:04.388579    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 09 21:40:14.388401 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:14.388361    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 09 21:40:24.388414 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:24.388369    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 09 21:40:34.389384 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:34.389344    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 09 21:40:44.388483 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:44.388441    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.65:8080: connect: connection refused"
Sep 09 21:40:54.389574 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:54.389544    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:40:55.442000 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:55.441964    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"]
Sep 09 21:40:55.442436 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:55.442215    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container" containerID="cri-o://1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a" gracePeriod=30
Sep 09 21:40:55.442436 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:55.442243    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kube-rbac-proxy" containerID="cri-o://4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7" gracePeriod=30
Sep 09 21:40:55.747163 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:55.747084    2580 generic.go:356] "Generic (PLEG): container finished" podID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerID="4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7" exitCode=2
Sep 09 21:40:55.747163 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:55.747153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" event={"ID":"34c76101-e4a6-4171-bc46-e3d94641cca6","Type":"ContainerDied","Data":"4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7"}
Sep 09 21:40:59.384890 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:40:59.384836    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.65:8643/healthz\": dial tcp 10.132.0.65:8643: connect: connection refused"
Sep 09 21:41:00.196782 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.196732    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:41:00.292106 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.292028    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34c76101-e4a6-4171-bc46-e3d94641cca6-proxy-tls\") pod \"34c76101-e4a6-4171-bc46-e3d94641cca6\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") "
Sep 09 21:41:00.292106 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.292071    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8x9s\" (UniqueName: \"kubernetes.io/projected/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-api-access-w8x9s\") pod \"34c76101-e4a6-4171-bc46-e3d94641cca6\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") "
Sep 09 21:41:00.292106 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.292093    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34c76101-e4a6-4171-bc46-e3d94641cca6-kserve-provision-location\") pod \"34c76101-e4a6-4171-bc46-e3d94641cca6\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") "
Sep 09 21:41:00.292398 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.292139    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-cabundle-cert\") pod \"34c76101-e4a6-4171-bc46-e3d94641cca6\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") "
Sep 09 21:41:00.292398 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.292196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-rbac-proxy-sar-config\") pod \"34c76101-e4a6-4171-bc46-e3d94641cca6\" (UID: \"34c76101-e4a6-4171-bc46-e3d94641cca6\") "
Sep 09 21:41:00.292499 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.292477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34c76101-e4a6-4171-bc46-e3d94641cca6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "34c76101-e4a6-4171-bc46-e3d94641cca6" (UID: "34c76101-e4a6-4171-bc46-e3d94641cca6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:41:00.292565 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.292525    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "34c76101-e4a6-4171-bc46-e3d94641cca6" (UID: "34c76101-e4a6-4171-bc46-e3d94641cca6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:41:00.292565 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.292550    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "34c76101-e4a6-4171-bc46-e3d94641cca6" (UID: "34c76101-e4a6-4171-bc46-e3d94641cca6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:41:00.294379 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.294348    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-api-access-w8x9s" (OuterVolumeSpecName: "kube-api-access-w8x9s") pod "34c76101-e4a6-4171-bc46-e3d94641cca6" (UID: "34c76101-e4a6-4171-bc46-e3d94641cca6"). InnerVolumeSpecName "kube-api-access-w8x9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:41:00.294492 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.294415    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34c76101-e4a6-4171-bc46-e3d94641cca6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "34c76101-e4a6-4171-bc46-e3d94641cca6" (UID: "34c76101-e4a6-4171-bc46-e3d94641cca6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:41:00.392818 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.392754    2580 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/34c76101-e4a6-4171-bc46-e3d94641cca6-proxy-tls\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:41:00.392818 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.392814    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8x9s\" (UniqueName: \"kubernetes.io/projected/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-api-access-w8x9s\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:41:00.392818 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.392823    2580 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/34c76101-e4a6-4171-bc46-e3d94641cca6-kserve-provision-location\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:41:00.393244 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.392832    2580 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-cabundle-cert\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:41:00.393244 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.392840    2580 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/34c76101-e4a6-4171-bc46-e3d94641cca6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-40.ec2.internal\" DevicePath \"\""
Sep 09 21:41:00.771711 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.771675    2580 generic.go:356] "Generic (PLEG): container finished" podID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerID="1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a" exitCode=0
Sep 09 21:41:00.771906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.771738    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"
Sep 09 21:41:00.771906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.771780    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" event={"ID":"34c76101-e4a6-4171-bc46-e3d94641cca6","Type":"ContainerDied","Data":"1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a"}
Sep 09 21:41:00.771906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.771847    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8" event={"ID":"34c76101-e4a6-4171-bc46-e3d94641cca6","Type":"ContainerDied","Data":"89cc8fcdafdd8f301ef3fda49fc31973068cf14d8deac3d5d365fd908823d7da"}
Sep 09 21:41:00.771906 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.771874    2580 scope.go:117] "RemoveContainer" containerID="4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7"
Sep 09 21:41:00.781265 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.781249    2580 scope.go:117] "RemoveContainer" containerID="1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a"
Sep 09 21:41:00.789077 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.789062    2580 scope.go:117] "RemoveContainer" containerID="451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2"
Sep 09 21:41:00.795682 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.795663    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"]
Sep 09 21:41:00.796898 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.796878    2580 scope.go:117] "RemoveContainer" containerID="4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7"
Sep 09 21:41:00.797160 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:41:00.797141    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7\": container with ID starting with 4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7 not found: ID does not exist" containerID="4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7"
Sep 09 21:41:00.797219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.797168    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7"} err="failed to get container status \"4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7\": rpc error: code = NotFound desc = could not find container \"4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7\": container with ID starting with 4cc14f9a5c0da10641710c9e8dd0a5e4de845891ae58788d75c4b17be997f8e7 not found: ID does not exist"
Sep 09 21:41:00.797219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.797184    2580 scope.go:117] "RemoveContainer" containerID="1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a"
Sep 09 21:41:00.797449 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:41:00.797432    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a\": container with ID starting with 1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a not found: ID does not exist" containerID="1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a"
Sep 09 21:41:00.797495 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.797454    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a"} err="failed to get container status \"1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a\": rpc error: code = NotFound desc = could not find container \"1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a\": container with ID starting with 1e693da88a4afd6b52d7bb4ed0822c9f5f6f60dcae268de3475f6e2a66c2cd9a not found: ID does not exist"
Sep 09 21:41:00.797495 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.797470    2580 scope.go:117] "RemoveContainer" containerID="451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2"
Sep 09 21:41:00.797693 ip-10-0-130-40 kubenswrapper[2580]: E0909 21:41:00.797677    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2\": container with ID starting with 451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2 not found: ID does not exist" containerID="451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2"
Sep 09 21:41:00.797733 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.797696    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2"} err="failed to get container status \"451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2\": rpc error: code = NotFound desc = could not find container \"451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2\": container with ID starting with 451a2d0344a402a338bf5ea5d395c7b8b2fba7dbbf295f41492a97464b60a7b2 not found: ID does not exist"
Sep 09 21:41:00.801267 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.801245    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-6f74d47cd4-5knp8"]
Sep 09 21:41:00.935723 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:41:00.935690    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" path="/var/lib/kubelet/pods/34c76101-e4a6-4171-bc46-e3d94641cca6/volumes"
Sep 09 21:42:05.170274 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:42:05.170167    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:42:05.174335 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:42:05.173946    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:47:05.201423 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:05.201315    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:47:05.205386 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:05.205025    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:47:29.447651 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:29.447613    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zdt9w_374b7dce-d3a8-4665-a0ad-96c74a838738/global-pull-secret-syncer/0.log"
Sep 09 21:47:29.588648 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:29.588614    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q2tpw_596b15b6-2d26-46db-a293-495852388c4f/konnectivity-agent/0.log"
Sep 09 21:47:29.611593 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:29.611564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-40.ec2.internal_16a910e8d25a6819bae738e4b0e06e46/haproxy/0.log"
Sep 09 21:47:33.833320 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:33.833290    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c397cbe0-3a68-4254-9e54-e9d0a5c4271f/alertmanager/0.log"
Sep 09 21:47:33.859956 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:33.859930    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c397cbe0-3a68-4254-9e54-e9d0a5c4271f/config-reloader/0.log"
Sep 09 21:47:33.884219 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:33.884198    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c397cbe0-3a68-4254-9e54-e9d0a5c4271f/kube-rbac-proxy-web/0.log"
Sep 09 21:47:33.914057 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:33.914030    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c397cbe0-3a68-4254-9e54-e9d0a5c4271f/kube-rbac-proxy/0.log"
Sep 09 21:47:33.952818 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:33.952796    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c397cbe0-3a68-4254-9e54-e9d0a5c4271f/kube-rbac-proxy-metric/0.log"
Sep 09 21:47:33.983168 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:33.983144    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c397cbe0-3a68-4254-9e54-e9d0a5c4271f/prom-label-proxy/0.log"
Sep 09 21:47:34.010358 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.010327    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c397cbe0-3a68-4254-9e54-e9d0a5c4271f/init-config-reloader/0.log"
Sep 09 21:47:34.098820 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.098792    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-z66wv_d834aab5-aae4-4735-a1a0-a375d3f37d09/cluster-monitoring-operator/0.log"
Sep 09 21:47:34.401908 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.401834    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2xtj_edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab/node-exporter/0.log"
Sep 09 21:47:34.422678 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.422646    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2xtj_edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab/kube-rbac-proxy/0.log"
Sep 09 21:47:34.444423 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.444393    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t2xtj_edaf66b9-6969-4d6c-8fcd-74cc7eecc5ab/init-textfile/0.log"
Sep 09 21:47:34.548876 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.548846    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pvq5c_e4985c20-bdee-4ef6-8485-23f67b083942/kube-rbac-proxy-main/0.log"
Sep 09 21:47:34.575002 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.574976    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pvq5c_e4985c20-bdee-4ef6-8485-23f67b083942/kube-rbac-proxy-self/0.log"
Sep 09 21:47:34.599073 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.599046    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-pvq5c_e4985c20-bdee-4ef6-8485-23f67b083942/openshift-state-metrics/0.log"
Sep 09 21:47:34.830025 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.829942    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rczwq_8dc74f72-44b5-49d1-9df3-ed4d978e2eb6/prometheus-operator/0.log"
Sep 09 21:47:34.854499 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.854478    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-rczwq_8dc74f72-44b5-49d1-9df3-ed4d978e2eb6/kube-rbac-proxy/0.log"
Sep 09 21:47:34.897460 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:34.897429    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-dlxf9_04cd56f3-59a3-4ffa-8318-18a418bbb463/prometheus-operator-admission-webhook/0.log"
Sep 09 21:47:37.381084 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.381050    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-8z2cm_c6ebbd15-7d72-4c09-a728-941a45f7af33/download-server/0.log"
Sep 09 21:47:37.872183 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.872149    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"]
Sep 09 21:47:37.872511 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.872498    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="storage-initializer"
Sep 09 21:47:37.872511 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.872512    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="storage-initializer"
Sep 09 21:47:37.872594 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.872521    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container"
Sep 09 21:47:37.872594 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.872527    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container"
Sep 09 21:47:37.872594 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.872533    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kube-rbac-proxy"
Sep 09 21:47:37.872594 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.872538    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kube-rbac-proxy"
Sep 09 21:47:37.872594 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.872594    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kube-rbac-proxy"
Sep 09 21:47:37.872754 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.872608    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34c76101-e4a6-4171-bc46-e3d94641cca6" containerName="kserve-container"
Sep 09 21:47:37.875630 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.875611    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:37.877983 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.877949    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qw289\"/\"openshift-service-ca.crt\""
Sep 09 21:47:37.878112 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.878063    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qw289\"/\"kube-root-ca.crt\""
Sep 09 21:47:37.878112 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.878063    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qw289\"/\"default-dockercfg-x8qhr\""
Sep 09 21:47:37.886914 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.886895    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"]
Sep 09 21:47:37.931515 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.931489    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-695c895bd6-7wc7h_814516c5-5564-4b9b-9753-883546d6093e/volume-data-source-validator/0.log"
Sep 09 21:47:37.960225 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.960195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-proc\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:37.960380 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.960242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-podres\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:37.960380 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.960262    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-sys\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:37.960380 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.960343    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-lib-modules\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:37.960518 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:37.960382    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lmcx\" (UniqueName: \"kubernetes.io/projected/09da64c0-9beb-48c4-a5b4-1dd97b919229-kube-api-access-8lmcx\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.060788 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.060728    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-lib-modules\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.060974 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.060799    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lmcx\" (UniqueName: \"kubernetes.io/projected/09da64c0-9beb-48c4-a5b4-1dd97b919229-kube-api-access-8lmcx\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.060974 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.060892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-proc\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.060974 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.060907    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-lib-modules\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.060974 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.060968    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-podres\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.061164 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.060993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-proc\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.061164 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.061005    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-sys\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.061164 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.061055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-sys\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.061164 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.061128    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/09da64c0-9beb-48c4-a5b4-1dd97b919229-podres\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.069302 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.069280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lmcx\" (UniqueName: \"kubernetes.io/projected/09da64c0-9beb-48c4-a5b4-1dd97b919229-kube-api-access-8lmcx\") pod \"perf-node-gather-daemonset-6b67m\" (UID: \"09da64c0-9beb-48c4-a5b4-1dd97b919229\") " pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.185947 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.185867    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.464241 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.464197    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"]
Sep 09 21:47:38.467262 ip-10-0-130-40 kubenswrapper[2580]: W0909 21:47:38.467231    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod09da64c0_9beb_48c4_a5b4_1dd97b919229.slice/crio-e9feb6d0ab9a4e2cb7347c800a93736190778cd258c96f21ac1b2f1505816674 WatchSource:0}: Error finding container e9feb6d0ab9a4e2cb7347c800a93736190778cd258c96f21ac1b2f1505816674: Status 404 returned error can't find the container with id e9feb6d0ab9a4e2cb7347c800a93736190778cd258c96f21ac1b2f1505816674
Sep 09 21:47:38.592893 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.592858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m" event={"ID":"09da64c0-9beb-48c4-a5b4-1dd97b919229","Type":"ContainerStarted","Data":"bf61739e6deb3bf6a7354b8ae0975718ce9b932a3cd5423f98a71679ff26b81f"}
Sep 09 21:47:38.593008 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.592916    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m" event={"ID":"09da64c0-9beb-48c4-a5b4-1dd97b919229","Type":"ContainerStarted","Data":"e9feb6d0ab9a4e2cb7347c800a93736190778cd258c96f21ac1b2f1505816674"}
Sep 09 21:47:38.593008 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.592989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:38.609941 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.609896    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m" podStartSLOduration=1.609877686 podStartE2EDuration="1.609877686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:47:38.609095243 +0000 UTC m=+3348.170122575" watchObservedRunningTime="2026-09-09 21:47:38.609877686 +0000 UTC m=+3348.170905004"
Sep 09 21:47:38.942335 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.942308    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fpc6w_e815f55d-a4cf-463f-8190-b8fe0f3bd98c/dns/0.log"
Sep 09 21:47:38.969025 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:38.968999    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fpc6w_e815f55d-a4cf-463f-8190-b8fe0f3bd98c/kube-rbac-proxy/0.log"
Sep 09 21:47:39.097528 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:39.097502    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xxnbh_4df75668-d74f-4dde-81dd-ecac0a82140a/dns-node-resolver/0.log"
Sep 09 21:47:39.681947 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:39.681916    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7zwsv_9b73b24e-5ef0-4299-90d8-b7932046333b/node-ca/0.log"
Sep 09 21:47:40.824541 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:40.824509    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fffxj_b817371a-9b4f-45c4-a96b-e4d22d616ca4/serve-healthcheck-canary/0.log"
Sep 09 21:47:41.256125 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:41.256090    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bthvc_1b67f017-f68f-4ced-85c4-01baa0fc1d0f/kube-rbac-proxy/0.log"
Sep 09 21:47:41.278135 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:41.278109    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bthvc_1b67f017-f68f-4ced-85c4-01baa0fc1d0f/exporter/0.log"
Sep 09 21:47:41.300990 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:41.300953    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bthvc_1b67f017-f68f-4ced-85c4-01baa0fc1d0f/extractor/0.log"
Sep 09 21:47:43.882049 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:43.882009    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-wd6sf_832d2964-8457-4251-9724-69c467c45a4b/manager/0.log"
Sep 09 21:47:43.903187 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:43.903165    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-78446_2daaf279-d2fe-4f35-bb59-2b38b27bf962/server/0.log"
Sep 09 21:47:44.406789 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.406740    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-58cfc_7e4947fe-03a9-4b6c-889e-3bc9c3d3a401/manager/0.log"
Sep 09 21:47:44.450015 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.449987    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pgtx9_e99f790f-82a3-4ef5-9231-31ea5a96f1b0/s3-init/0.log"
Sep 09 21:47:44.471584 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.471553    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-pgtx9_e99f790f-82a3-4ef5-9231-31ea5a96f1b0/download-hf-model/0.log"
Sep 09 21:47:44.505621 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.505593    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-vh68l_1d4dac54-92a9-483e-940a-cc609a72c94e/s3-tls-init-custom/0.log"
Sep 09 21:47:44.562802 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.562748    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-vh68l_1d4dac54-92a9-483e-940a-cc609a72c94e/download-hf-model/0.log"
Sep 09 21:47:44.591918 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.591883    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-hgdvj_0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2/s3-tls-init-serving/0.log"
Sep 09 21:47:44.609201 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.609178    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qw289/perf-node-gather-daemonset-6b67m"
Sep 09 21:47:44.610679 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.610658    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-hgdvj_0a2cf039-eb5b-4f52-a0a6-aeaf0c91d6e2/download-hf-model/0.log"
Sep 09 21:47:44.642571 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.642536    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-wqfzk_efadc29a-a7e9-4adc-81df-2ba658b665c9/seaweedfs/0.log"
Sep 09 21:47:44.669525 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.669450    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-sb57f_c2f94f01-0623-4a8b-8fb0-309ca75d6725/seaweedfs-tls-custom/0.log"
Sep 09 21:47:44.694411 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:44.694388    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-9rcb7_da71c3f6-3df0-4dcc-8203-6cefc4023785/seaweedfs-tls-serving/0.log"
Sep 09 21:47:49.688470 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:49.688360    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-mnx76_04dbd6bf-11a8-4786-83d2-6e8df4706cfd/kube-storage-version-migrator-operator/1.log"
Sep 09 21:47:49.690572 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:49.690546    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-8d7dd4675-mnx76_04dbd6bf-11a8-4786-83d2-6e8df4706cfd/kube-storage-version-migrator-operator/0.log"
Sep 09 21:47:50.613292 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:50.613262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6phgb_f3b6ca73-058e-4a22-9cfa-dbbf72f9564b/kube-multus-additional-cni-plugins/0.log"
Sep 09 21:47:50.649150 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:50.649125    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6phgb_f3b6ca73-058e-4a22-9cfa-dbbf72f9564b/egress-router-binary-copy/0.log"
Sep 09 21:47:50.674355 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:50.674329    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6phgb_f3b6ca73-058e-4a22-9cfa-dbbf72f9564b/cni-plugins/0.log"
Sep 09 21:47:50.701403 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:50.701380    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6phgb_f3b6ca73-058e-4a22-9cfa-dbbf72f9564b/bond-cni-plugin/0.log"
Sep 09 21:47:50.723512 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:50.723490    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6phgb_f3b6ca73-058e-4a22-9cfa-dbbf72f9564b/routeoverride-cni/0.log"
Sep 09 21:47:50.755526 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:50.755502    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6phgb_f3b6ca73-058e-4a22-9cfa-dbbf72f9564b/whereabouts-cni-bincopy/0.log"
Sep 09 21:47:50.806951 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:50.806924    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6phgb_f3b6ca73-058e-4a22-9cfa-dbbf72f9564b/whereabouts-cni/0.log"
Sep 09 21:47:51.381532 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:51.381500    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wh7mr_fe5b2e20-19a6-4742-8f06-c1f10f896bb4/kube-multus/0.log"
Sep 09 21:47:51.529413 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:51.529382    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nzvfm_b7b1474e-02e4-4041-a48a-2a3a652388c9/network-metrics-daemon/0.log"
Sep 09 21:47:51.550665 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:51.550633    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-nzvfm_b7b1474e-02e4-4041-a48a-2a3a652388c9/kube-rbac-proxy/0.log"
Sep 09 21:47:52.490152 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:52.490114    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-controller/0.log"
Sep 09 21:47:52.566499 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:52.566468    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/1.log"
Sep 09 21:47:52.566499 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:52.566493    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovn-acl-logging/0.log"
Sep 09 21:47:52.589293 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:52.589262    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/kube-rbac-proxy-node/0.log"
Sep 09 21:47:52.617030 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:52.617001    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/kube-rbac-proxy-ovn-metrics/0.log"
Sep 09 21:47:52.639515 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:52.639490    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/northd/0.log"
Sep 09 21:47:52.661404 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:52.661371    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/nbdb/0.log"
Sep 09 21:47:52.683583 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:52.683564    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/sbdb/0.log"
Sep 09 21:47:52.745812 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:52.745726    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-fc6xb_326bde36-e763-476f-92d8-e411d84cad01/ovnkube-controller/0.log"
Sep 09 21:47:54.286448 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:54.286370    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-qd7jf_fcc45c41-2627-41ac-97fa-7baaa2cce355/check-endpoints/0.log"
Sep 09 21:47:54.378932 ip-10-0-130-40 kubenswrapper[2580]: I0909 21:47:54.378902    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-xwlqv_45839b82-7bb9-4f27-8efe-89affa6fc06a/network-check-target-container/0.log"