Jul 28 17:10:41.236620 ip-10-0-139-190 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:10:41.637231 ip-10-0-139-190 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:41.637231 ip-10-0-139-190 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:10:41.637231 ip-10-0-139-190 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:41.637231 ip-10-0-139-190 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:10:41.637231 ip-10-0-139-190 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:41.638277 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.638175    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:10:41.644570 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644552    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:41.644570 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644568    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644574    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644577    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644580    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644583    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644586    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644588    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644592    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644595    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644597    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644602    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644607    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644609    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644612    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644615    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644617    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644620    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644622    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644625    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:41.644642 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644627    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644630    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644633    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644635    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644638    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644641    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644645    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644649    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644652    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644655    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644658    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644661    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644664    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644666    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644669    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644672    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644675    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644677    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644683    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644686    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:41.645103 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644688    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644691    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644693    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644696    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644699    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644701    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644703    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644706    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644708    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644711    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644713    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644716    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644718    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644721    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644726    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644729    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644731    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644734    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644737    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644740    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:41.645610 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644743    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644745    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644748    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644750    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644753    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644755    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644758    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644760    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644763    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644766    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644768    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644773    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644776    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644778    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644781    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644783    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644786    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644788    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644791    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644793    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:41.646139 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644796    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644798    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644801    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644803    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644807    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644810    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644813    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644815    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644818    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644821    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644824    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644828    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644831    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644834    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644837    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644840    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644843    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644845    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.644848    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:41.646641 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645560    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645567    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645570    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645573    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645576    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645580    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645582    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645585    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645588    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645590    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645593    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645595    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645598    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645600    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645604    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645606    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645609    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645612    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645615    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:41.647104 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645619    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645622    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645624    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645627    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645630    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645632    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645635    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645637    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645640    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645643    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645645    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645648    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645651    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645653    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645656    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645660    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645663    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645666    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645669    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:41.647591 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645672    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645675    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645678    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645680    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645683    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645686    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645688    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645691    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645693    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645696    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645700    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645703    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645706    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645709    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645712    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645714    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645717    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645720    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645722    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:41.648059 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645725    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645727    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645730    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645734    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645738    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645741    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645744    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645747    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645749    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645752    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645755    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645758    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645761    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645764    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645767    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645770    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645773    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645775    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645778    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645780    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:41.648554 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645785    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645787    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645790    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645793    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645796    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645799    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645801    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645804    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645806    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645809    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645812    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645814    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645817    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645820    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645822    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645825    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645827    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645830    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645832    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645835    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:41.649041 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645837    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.645840    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646231    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646236    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646239    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646255    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646259    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646261    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646264    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646267    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646271    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646275    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646280    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646282    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646285    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646288    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646291    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646294    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646297    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:41.649558 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646300    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646303    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646305    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646308    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646311    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646314    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646317    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646320    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646322    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646325    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646327    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646330    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646332    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646335    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646337    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646340    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646343    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646346    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646348    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646350    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:41.650033 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646355    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646358    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646360    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646363    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646367    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646371    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646374    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646377    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646380    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646383    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646386    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646388    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646391    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646394    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646396    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646399    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646402    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646404    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646406    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:41.650533 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646409    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646411    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646414    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646416    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646419    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646421    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646424    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646426    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646429    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646431    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646434    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646436    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646438    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646441    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646444    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646447    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646449    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646452    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646456    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646458    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:41.651021 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646461    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646463    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646467    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646469    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646472    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646475    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646477    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646480    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646483    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646485    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646488    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646490    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646493    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646496    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646498    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646501    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646505    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646508    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646510    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646513    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:41.651567 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646516    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646518    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.646521    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646601    2577 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646615    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646622    2577 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646626    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646631    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646634    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646638    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646644    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646647    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646650    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646654    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646660    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646663    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646666    2577 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646668    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646671    2577 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646674    2577 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646677    2577 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646681    2577 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646684    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646687    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:10:41.652046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646690    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646694    2577 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646697    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646700    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646703    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646706    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646709    2577 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646712    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646715    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646719    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646722    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646726    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646729    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646732    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646735    2577 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646738    2577 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646742    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646746    2577 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646750    2577 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646753    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646756    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646759    2577 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646765    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646768    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646771    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:10:41.652644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646774    2577 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646777    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646780    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646783    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646786    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646788    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646791    2577 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646794    2577 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646798    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646801    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646804    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646807    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646809    2577 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646812    2577 flags.go:64] FLAG: --help="false"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646815    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-190.ec2.internal"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646818    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646821    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646824    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646827    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646830    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646833    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646836    2577 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646839    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:10:41.653383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646841    2577 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646845    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646848    2577 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646851    2577 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646854    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646857    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646862    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646865    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646867    2577 flags.go:64] FLAG: --lock-file=""
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646870    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646873    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646876    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646882    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646885    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646887    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646890    2577 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646893    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646896    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646899    2577 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646901    2577 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646905    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646909    2577 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646913    2577 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646915    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646918    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:10:41.654058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646921    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646924    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646927    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646930    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646933    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646940    2577 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646943    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646946    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646949    2577 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646953    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646959    2577 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646962    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646965    2577 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646969    2577 flags.go:64] FLAG: --port="10250"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646972    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646976    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0256efcd5e862dd6f"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646979    2577 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646982    2577 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646985    2577 flags.go:64] FLAG: --register-node="true"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646988    2577 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646992    2577 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646994    2577 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.646997    2577 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647000    2577 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647006    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:10:41.654811 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647009    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647012    2577 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647015    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647018    2577 flags.go:64] FLAG: --runonce="false"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647021    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647024    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647026    2577 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647029    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647032    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647035    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647038    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647041    2577 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647044    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647047    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647050    2577 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647052    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647055    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647059    2577 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647062    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647067    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647071    2577 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647074    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647078    2577 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647081    2577 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647084    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:10:41.655534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647087    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647089    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647092    2577 flags.go:64] FLAG: --v="2"
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647096    2577 flags.go:64] FLAG: --version="false"
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647100    2577 flags.go:64] FLAG: --vmodule=""
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647104    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647108    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647220    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647224    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647228    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647231    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647233    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647236    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647239    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647256    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647259    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647262    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647265    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647267    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647270    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647273    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:41.656226 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647275    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647278    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647280    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647285    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647288    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647291    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647295    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647297    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647300    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647302    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647305    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647308    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647310    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647313    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647315    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647318    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647320    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647323    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647325    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647328    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:41.656819 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647331    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647333    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647336    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647339    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647341    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647344    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647346    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647349    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647351    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647354    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647356    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647359    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647361    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647364    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647366    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647370    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647373    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647376    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647380    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647383    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:41.657489 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647386    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647388    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647391    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647393    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647396    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647399    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647401    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647404    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647406    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647408    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647411    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647413    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647416    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647420    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647423    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647426    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647429    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647432    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647435    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:41.658182 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647438    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647441    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647444    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647447    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647449    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647452    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647455    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647458    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647462    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647465    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647467    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647476    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647480    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647483    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647486    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647490    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647493    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647496    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647498    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:41.658755 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647501    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647503    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647506    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647508    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647511    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647513    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.647516    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.647521    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.654525    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.654541    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.655312    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.655319    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655378    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655384    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655387    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:41.659301 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655391    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655394    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655397    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655400    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655403    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655406    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655410    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655413    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655415    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655418    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655420    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655423    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655426    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655428    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655431    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655433    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655436    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655438    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655441    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655443    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:41.659718 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655446    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655449    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655451    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655454    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655457    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655460    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655464    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655468    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655472    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655475    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655478    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655480    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655483    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655486    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655490    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655494    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655497    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655500    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655502    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:41.660272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655505    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655508    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655510    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655513    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655515    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655518    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655522    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655525    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655528    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655531    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655534    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655537    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655540    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655544    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655547    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655550    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655554    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655557    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655560    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:41.660795 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655562    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655566    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655569    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655571    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655574    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655576    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655579    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655582    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655584    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655587    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655589    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655592    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655594    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655597    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655599    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655602    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655604    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655607    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655609    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655612    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:41.661332 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655614    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655617    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655619    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655622    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655624    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655626    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655629    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655631    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655634    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655637    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655639    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655642    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655645    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655647    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655650    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655653    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655656    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:41.661869 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655658    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.655663    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655777    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655783    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655786    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655788    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655791    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655794    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655796    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655799    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655801    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655804    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655806    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655809    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655812    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:41.662391 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655815    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655820    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655824    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655827    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655830    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655833    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655836    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655839    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655841    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655844    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655848    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655851    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655854    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655856    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655859    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655862    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655866    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655869    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655872    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:41.662773 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655875    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655877    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655880    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655883    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655886    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655888    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655891    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655894    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655896    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655899    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655902    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655904    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655907    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655909    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655912    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655914    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655917    2577 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655919    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655921    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655924    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:41.663292 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655926    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655929    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655931    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655934    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655937    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655940    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655942    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655945    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655947    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655950    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655953    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655955    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655958    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655960    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655963    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655965    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655968    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655970    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655974    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655977    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:41.663776 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655980    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655983    2577 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655986    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655988    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655991    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655994    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655997    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.655999    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656002    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656004    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656007    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656010    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656012    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656015    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656017    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656019    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656022    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656025    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656028    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656030    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:41.664272 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656033    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656035    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656038    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656041    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656043    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656046    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:41.656048    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.656053    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.656166    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.660885    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.662360    2577 server.go:1019] "Starting client certificate rotation"
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.662465    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:41.664805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.662505    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:41.666436 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.666422    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:41.667600 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.667582    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:41.678628 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.678603    2577 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:10:41.680736 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.680722    2577 log.go:25] "Validated CRI v1 image API"
Jul 28 17:10:41.682300 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.682286    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:10:41.684896 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.684875    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 af15f957-293e-4528-81ed-f74e18b353d5:/dev/nvme0n1p3 cd1b69ee-ee26-41b9-ac2f-2312548db02d:/dev/nvme0n1p4]
Jul 28 17:10:41.684954 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.684895    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:10:41.689993 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.689884    2577 manager.go:216] Machine: {Timestamp:2026-07-28 17:10:41.688597973 +0000 UTC m=+0.332567453 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120850 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d2dd845c175964c72c2e7b2d305a4 SystemUUID:ec2d2dd8-45c1-7596-4c72-c2e7b2d305a4 BootID:9e8c46fc-de35-4989-8b0d-e65e446734e6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9d:3d:1c:20:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9d:3d:1c:20:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:fe:0a:f0:5c:f2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:10:41.689993 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.689985    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:10:41.690155 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.690070    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:10:41.693173 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.693144    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:10:41.693372 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.693175    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:10:41.694048 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.694035    2577 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:10:41.694109 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.694053    2577 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:10:41.694109 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.694064    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:41.694821 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.694808    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:41.696281 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.696264    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:41.696397 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.696384    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:10:41.699540 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.699525    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:41.699673 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.699663    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:10:41.699727 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.699701    2577 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:10:41.699727 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.699716    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:10:41.699822 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.699735    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:10:41.699822 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.699749    2577 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:10:41.699822 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.699761    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:10:41.700165 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.700152    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:41.700234 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.700176    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:41.703407 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.703348    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:10:41.705140 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.705122    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:10:41.705215 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.705162    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:10:41.705932 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.705918    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:10:41.705996 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.705938    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:10:41.705996 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.705947    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:10:41.705996 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.705955    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:10:41.705996 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.705967    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:10:41.705996 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.705976    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:10:41.706798 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.706784    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:10:41.706867 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.706799    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:10:41.706867 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.706811    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:10:41.706867 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.706819    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:10:41.706867 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.706831    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:10:41.706867 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.706845    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:10:41.707774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.707762    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:10:41.707853 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.707775    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:10:41.711010 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.710995    2577 server.go:1268] "Started kubelet"
Jul 28 17:10:41.711128 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.711105    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:10:41.711217 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.711095    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:10:41.711217 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.711172    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:10:41.711217 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.711106    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:10:41.711919 ip-10-0-139-190 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:10:41.712455 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.712357    2577 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:10:41.713190 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.713168    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:10:41.713271 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.713171    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:10:41.713372 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.713356    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:10:41.716497 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.716474    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:10:41.720658 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.720639    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:10:41.720658 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.720652    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:41.721137 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.720780    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:41.721723 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721567    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:10:41.721723 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721581    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:10:41.721723 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721665    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:10:41.721723 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721685    2577 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:10:41.721723 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721692    2577 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:10:41.721970 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721758    2577 factory.go:153] Registering CRI-O factory
Jul 28 17:10:41.721970 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721771    2577 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:10:41.721970 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721897    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:10:41.721970 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721909    2577 factory.go:55] Registering systemd factory
Jul 28 17:10:41.721970 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721925    2577 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:10:41.721970 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721969    2577 factory.go:103] Registering Raw factory
Jul 28 17:10:41.722325 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.721985    2577 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:10:41.722707 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.722684    2577 manager.go:349] Starting recovery of all containers
Jul 28 17:10:41.726872 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.726799    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:10:41.728233 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.727175    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-190.ec2.internal.18c68346a360edf2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-190.ec2.internal,UID:ip-10-0-139-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-190.ec2.internal,},FirstTimestamp:2026-07-28 17:10:41.710968306 +0000 UTC m=+0.354937802,LastTimestamp:2026-07-28 17:10:41.710968306 +0000 UTC m=+0.354937802,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-190.ec2.internal,}"
Jul 28 17:10:41.732862 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.732689    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:10:41.733944 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.733919    2577 manager.go:354] Recovery completed
Jul 28 17:10:41.737521 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.737503    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n28zd"
Jul 28 17:10:41.739003 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.738990    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:41.741047 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.741031    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:41.741136 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.741061    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:41.741136 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.741073    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:41.741546 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.741535    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:41.741592 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.741532    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:10:41.741592 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.741567    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:10:41.741592 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.741584    2577 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:41.742205 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.742191    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:41.742328 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.742220    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:41.742328 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.742234    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:41.742815 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.742802    2577 policy_none.go:49] "None policy: Start"
Jul 28 17:10:41.742855 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.742819    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:10:41.742855 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.742829    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:10:41.743820 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.743805    2577 policy_none.go:47] "Start"
Jul 28 17:10:41.743943 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.743869    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-190.ec2.internal.18c68346a52be11c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-190.ec2.internal,UID:ip-10-0-139-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-190.ec2.internal,},FirstTimestamp:2026-07-28 17:10:41.741046044 +0000 UTC m=+0.385015525,LastTimestamp:2026-07-28 17:10:41.741046044 +0000 UTC m=+0.385015525,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-190.ec2.internal,}"
Jul 28 17:10:41.745926 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.745909    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n28zd"
Jul 28 17:10:41.780067 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.780040    2577 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:10:41.780212 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.780077    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:10:41.780212 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.780090    2577 server.go:87] "Starting device plugin registration server"
Jul 28 17:10:41.780410 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.780398    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:10:41.780455 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.780414    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:10:41.780548 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.780522    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:10:41.780654 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.780627    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:10:41.780654 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.780637    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:10:41.781167 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.781145    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:10:41.781277 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.781198    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:41.881871 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.881822    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:41.882929 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.882908    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:41.883034 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.882943    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:41.883034 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.882955    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:41.883034 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.882986    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-190.ec2.internal"
Jul 28 17:10:41.892693 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.892642    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-190.ec2.internal"
Jul 28 17:10:41.892693 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.892671    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-190.ec2.internal\": node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:41.897291 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.897265    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:10:41.898705 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.898679    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:10:41.898784 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.898722    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:10:41.898784 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.898743    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:10:41.898784 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.898775    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:10:41.901531 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:41.901514    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:41.914399 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:41.914380    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.000970 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.000914    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"]
Jul 28 17:10:42.001158 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.001047    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:42.002099 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.002084    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:42.002153 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.002115    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:42.002153 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.002126    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:42.003611 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.003595    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:42.003703 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.003688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.003754 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.003718    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:42.004378 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.004357    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:42.004491 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.004389    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:42.004491 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.004399    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:42.004491 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.004357    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:42.004491 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.004472    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:42.004491 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.004486    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:42.005655 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.005640    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.005713 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.005667    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:42.006395 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.006374    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:42.006513 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.006405    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:42.006513 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.006416    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:42.015211 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.015193    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.022677 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.022659    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9646f759e4901fd6d074052a7efe1831-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"9646f759e4901fd6d074052a7efe1831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.022739 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.022684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eee64e253de37dcdd5d1e7fdc42d07fa-config\") pod \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" (UID: \"eee64e253de37dcdd5d1e7fdc42d07fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.022739 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.022700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9646f759e4901fd6d074052a7efe1831-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"9646f759e4901fd6d074052a7efe1831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.033301 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.033280    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-190.ec2.internal\" not found" node="ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.038189 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.038171    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-190.ec2.internal\" not found" node="ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.115783 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.115747    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.123433 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.123408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9646f759e4901fd6d074052a7efe1831-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"9646f759e4901fd6d074052a7efe1831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.123498 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.123440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eee64e253de37dcdd5d1e7fdc42d07fa-config\") pod \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" (UID: \"eee64e253de37dcdd5d1e7fdc42d07fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.123498 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.123476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9646f759e4901fd6d074052a7efe1831-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"9646f759e4901fd6d074052a7efe1831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.123565 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.123505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9646f759e4901fd6d074052a7efe1831-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"9646f759e4901fd6d074052a7efe1831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.123565 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.123524    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eee64e253de37dcdd5d1e7fdc42d07fa-config\") pod \"kube-apiserver-proxy-ip-10-0-139-190.ec2.internal\" (UID: \"eee64e253de37dcdd5d1e7fdc42d07fa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.123565 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.123511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9646f759e4901fd6d074052a7efe1831-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal\" (UID: \"9646f759e4901fd6d074052a7efe1831\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.216666 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.216574    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.324371 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.324319    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.335403 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.335380    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.342144 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.342122    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:42.424514 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.424455    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.528606 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.528522    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.612991 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.612959    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:42.629464 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.629435    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.663196 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.663159    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:10:42.663790 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.663365    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:42.663790 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.663370    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:42.730168 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.730134    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.749881 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.749833    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:05:42 +0000 UTC" deadline="2028-01-31 02:32:32.51964087 +0000 UTC"
Jul 28 17:10:42.749881 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.749870    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13233h21m49.769774768s"
Jul 28 17:10:42.830922 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.830844    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.926477 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:42.926448    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9646f759e4901fd6d074052a7efe1831.slice/crio-46c80a0ef58b11537fc388c406eef38935743f2152a2ca8b95efba7c08159d65 WatchSource:0}: Error finding container 46c80a0ef58b11537fc388c406eef38935743f2152a2ca8b95efba7c08159d65: Status 404 returned error can't find the container with id 46c80a0ef58b11537fc388c406eef38935743f2152a2ca8b95efba7c08159d65
Jul 28 17:10:42.926785 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.926769    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:42.930911 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:42.930889    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:42.931647 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:42.931634    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:10:43.031104 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.031050    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:43.131760 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.131676    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:43.232530 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.232495    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-190.ec2.internal\" not found"
Jul 28 17:10:43.287774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.287745    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:43.309543 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:43.309518    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee64e253de37dcdd5d1e7fdc42d07fa.slice/crio-ab12e5915ee23a5f8424657ab94cd376f14d9de1295c0fda2b37751528706b5d WatchSource:0}: Error finding container ab12e5915ee23a5f8424657ab94cd376f14d9de1295c0fda2b37751528706b5d: Status 404 returned error can't find the container with id ab12e5915ee23a5f8424657ab94cd376f14d9de1295c0fda2b37751528706b5d
Jul 28 17:10:43.321566 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.321530    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:43.333715 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.333695    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:43.334614 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.334600    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal"
Jul 28 17:10:43.348687 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.348667    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:43.701843 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.701808    2577 apiserver.go:52] "Watching apiserver"
Jul 28 17:10:43.711590 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.711559    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:10:43.711933 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.711907    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-ck9nr","kube-system/global-pull-secret-syncer-c6vbq","kube-system/konnectivity-agent-vvkr9","openshift-cluster-node-tuning-operator/tuned-fk7jp","openshift-image-registry/node-ca-jht5g","openshift-multus/network-metrics-daemon-6czkd","openshift-network-operator/iptables-alerter-5m4kt","kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5","openshift-dns/node-resolver-mz5qt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal","openshift-multus/multus-9wfsw","openshift-multus/multus-additional-cni-plugins-j7k5l","openshift-network-diagnostics/network-check-target-b5btq"]
Jul 28 17:10:43.713521 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.713497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.714776 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.714755    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.715930 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.715868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:10:43.716541 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.716492    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:43.716641 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.716598    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:10:43.716867 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.716850    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lx6v9\""
Jul 28 17:10:43.717183 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.717166    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:43.717263 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.717174    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.718387 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.718040    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:10:43.718387 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.718051    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:10:43.718387 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.718095    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:10:43.718387 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.718202    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ngkq2\""
Jul 28 17:10:43.718387 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.718043    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:10:43.718663 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.718403    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:10:43.718711 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.718689    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.719367 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.719347    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qls8h\""
Jul 28 17:10:43.719456 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.719356    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:10:43.719788 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.719766    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:43.719889 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.719857    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:10:43.720268 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.720179    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wdg2v\""
Jul 28 17:10:43.720268 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.720182    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:43.720268 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.720212    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:43.720268 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.720268    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:10:43.721288 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.721271    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:10:43.721599 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.721576    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:10:43.721725 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.721610    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-q488b\""
Jul 28 17:10:43.721813 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.721749    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:10:43.721973 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.721955    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:43.723384 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.723364    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.725173 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.725151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.725886 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.725868    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jhccw\""
Jul 28 17:10:43.726207 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.726164    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:10:43.726427 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.726407    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:10:43.726518 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.726448    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:10:43.726659 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.726646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:43.726716 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.726699    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:10:43.728007 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.727927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:10:43.728007 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.727946    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:10:43.728252 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.728232    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k56zz\""
Jul 28 17:10:43.728377 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.728287    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.728665 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.728649    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:10:43.729373 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.729353    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:10:43.729473 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.729354    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:10:43.729615 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.729597    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:10:43.730388 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.730367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:43.730484 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.730449    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:10:43.730915 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.730895    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:10:43.731239 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.731219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fww8f\""
Jul 28 17:10:43.731434 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.731363    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:10:43.732929 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.732892    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.735090 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-lib-modules\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.735090 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c023b43e-30f9-4d56-aa4f-50f5e86724cf-serviceca\") pod \"node-ca-jht5g\" (UID: \"c023b43e-30f9-4d56-aa4f-50f5e86724cf\") " pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.735285 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-registration-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.735285 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-device-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.735285 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b7cd99b-bfab-4d37-a1b4-9d39302ba332-host-slash\") pod \"iptables-alerter-5m4kt\" (UID: \"8b7cd99b-bfab-4d37-a1b4-9d39302ba332\") " pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.735285 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735209    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-modprobe-d\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.735285 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-sysctl-d\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.735285 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c023b43e-30f9-4d56-aa4f-50f5e86724cf-host\") pod \"node-ca-jht5g\" (UID: \"c023b43e-30f9-4d56-aa4f-50f5e86724cf\") " pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.735582 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735289    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-etc-selinux\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.735582 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735315    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qx8\" (UniqueName: \"kubernetes.io/projected/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-kube-api-access-l2qx8\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.735582 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735360    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw7tb\" (UniqueName: \"kubernetes.io/projected/8b7cd99b-bfab-4d37-a1b4-9d39302ba332-kube-api-access-dw7tb\") pod \"iptables-alerter-5m4kt\" (UID: \"8b7cd99b-bfab-4d37-a1b4-9d39302ba332\") " pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.736454 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.736428    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:10:43.737111 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.737085    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xxghb\""
Jul 28 17:10:43.737744 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.735405    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-system-cni-dir\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.737841 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.737782    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/facca3dd-945a-42d5-a32a-b8e4de3f7108-tmp\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.737841 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.737817    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs8gb\" (UniqueName: \"kubernetes.io/projected/c023b43e-30f9-4d56-aa4f-50f5e86724cf-kube-api-access-hs8gb\") pod \"node-ca-jht5g\" (UID: \"c023b43e-30f9-4d56-aa4f-50f5e86724cf\") " pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.737997 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.737845    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-kubelet-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.737997 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.737874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2a29023-9a19-4533-afcf-b3ea0ac28e63-agent-certs\") pod \"konnectivity-agent-vvkr9\" (UID: \"e2a29023-9a19-4533-afcf-b3ea0ac28e63\") " pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:10:43.738092 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6vww\" (UniqueName: \"kubernetes.io/projected/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-kube-api-access-m6vww\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.738092 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738065    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-kubernetes\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.738190 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-sysctl-conf\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.738242 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738128    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b7cd99b-bfab-4d37-a1b4-9d39302ba332-iptables-alerter-script\") pod \"iptables-alerter-5m4kt\" (UID: \"8b7cd99b-bfab-4d37-a1b4-9d39302ba332\") " pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.738293 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738254    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.738384 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738288    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.738384 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-sysconfig\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.738484 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-var-lib-kubelet\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.738484 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738465    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-host\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.738578 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt4qf\" (UniqueName: \"kubernetes.io/projected/facca3dd-945a-42d5-a32a-b8e4de3f7108-kube-api-access-nt4qf\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.738578 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738565    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-sys-fs\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.738678 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.738678 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.738774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738695    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-sys\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.738774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738719    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:43.738774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738739    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2a29023-9a19-4533-afcf-b3ea0ac28e63-konnectivity-ca\") pod \"konnectivity-agent-vvkr9\" (UID: \"e2a29023-9a19-4533-afcf-b3ea0ac28e63\") " pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:10:43.738774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738766    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-run\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.738960 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-tuned\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.738960 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29nv\" (UniqueName: \"kubernetes.io/projected/9cdd97a5-b1fd-4756-9933-1a551767c9c6-kube-api-access-t29nv\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:43.738960 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-socket-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.738960 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-cnibin\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.738960 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-systemd\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.739185 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.738959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-os-release\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.739185 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.739077    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:43.763431 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.763404    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vq9rz"
Jul 28 17:10:43.773162 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.773135    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vq9rz"
Jul 28 17:10:43.822954 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.822924    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:10:43.839734 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839701    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-lib-modules\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.839879 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b7cd99b-bfab-4d37-a1b4-9d39302ba332-host-slash\") pod \"iptables-alerter-5m4kt\" (UID: \"8b7cd99b-bfab-4d37-a1b4-9d39302ba332\") " pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.839879 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839775    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a0a67e1-6d79-423b-8995-cd624597642f-kubelet-config\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:43.839879 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839804    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7cc17ed-e2a2-4226-9287-63c1c56238fc-ovnkube-config\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.839879 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839829    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7cc17ed-e2a2-4226-9287-63c1c56238fc-env-overrides\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.839879 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839853    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-etc-kubernetes\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.839879 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8b7cd99b-bfab-4d37-a1b4-9d39302ba332-host-slash\") pod \"iptables-alerter-5m4kt\" (UID: \"8b7cd99b-bfab-4d37-a1b4-9d39302ba332\") " pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.839879 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-etc-selinux\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qx8\" (UniqueName: \"kubernetes.io/projected/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-kube-api-access-l2qx8\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-lib-modules\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-os-release\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-etc-selinux\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.839973    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-kubelet-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840014    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-run-netns\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840114    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-node-log\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-log-socket\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c40a55bc-b4ed-4df5-8c3c-a939e7989b12-hosts-file\") pod \"node-resolver-mz5qt\" (UID: \"c40a55bc-b4ed-4df5-8c3c-a939e7989b12\") " pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840227    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-system-cni-dir\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-kubelet-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b7cd99b-bfab-4d37-a1b4-9d39302ba332-iptables-alerter-script\") pod \"iptables-alerter-5m4kt\" (UID: \"8b7cd99b-bfab-4d37-a1b4-9d39302ba332\") " pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.840264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-run-openvswitch\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7cc17ed-e2a2-4226-9287-63c1c56238fc-ovn-node-metrics-cert\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-cni-dir\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-run-k8s-cni-cncf-io\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840350    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-conf-dir\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-sysconfig\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-var-lib-kubelet\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-host\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840449    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt4qf\" (UniqueName: \"kubernetes.io/projected/facca3dd-945a-42d5-a32a-b8e4de3f7108-kube-api-access-nt4qf\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-hostroot\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-tuned\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29nv\" (UniqueName: \"kubernetes.io/projected/9cdd97a5-b1fd-4756-9933-1a551767c9c6-kube-api-access-t29nv\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-socket-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840579    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840604    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c023b43e-30f9-4d56-aa4f-50f5e86724cf-serviceca\") pod \"node-ca-jht5g\" (UID: \"c023b43e-30f9-4d56-aa4f-50f5e86724cf\") " pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840625    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-registration-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.841125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-device-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840691    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840709    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-run-multus-certs\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-modprobe-d\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-sysctl-d\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840767    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c023b43e-30f9-4d56-aa4f-50f5e86724cf-host\") pod \"node-ca-jht5g\" (UID: \"c023b43e-30f9-4d56-aa4f-50f5e86724cf\") " pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw7tb\" (UniqueName: \"kubernetes.io/projected/8b7cd99b-bfab-4d37-a1b4-9d39302ba332-kube-api-access-dw7tb\") pod \"iptables-alerter-5m4kt\" (UID: \"8b7cd99b-bfab-4d37-a1b4-9d39302ba332\") " pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-system-cni-dir\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840826    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-var-lib-kubelet\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840846    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/facca3dd-945a-42d5-a32a-b8e4de3f7108-tmp\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs8gb\" (UniqueName: \"kubernetes.io/projected/c023b43e-30f9-4d56-aa4f-50f5e86724cf-kube-api-access-hs8gb\") pod \"node-ca-jht5g\" (UID: \"c023b43e-30f9-4d56-aa4f-50f5e86724cf\") " pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2a29023-9a19-4533-afcf-b3ea0ac28e63-agent-certs\") pod \"konnectivity-agent-vvkr9\" (UID: \"e2a29023-9a19-4533-afcf-b3ea0ac28e63\") " pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840926    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8b7cd99b-bfab-4d37-a1b4-9d39302ba332-iptables-alerter-script\") pod \"iptables-alerter-5m4kt\" (UID: \"8b7cd99b-bfab-4d37-a1b4-9d39302ba332\") " pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.840930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6vww\" (UniqueName: \"kubernetes.io/projected/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-kube-api-access-m6vww\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841004    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-etc-openvswitch\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-run-ovn\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841060    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-cni-netd\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.842006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-cni-binary-copy\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-kubernetes\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-sysctl-conf\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-sysconfig\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841224    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-kubelet\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-var-lib-kubelet\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841280    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-run-systemd\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-host\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-var-lib-cni-bin\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841349    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-sys-fs\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a0a67e1-6d79-423b-8995-cd624597642f-dbus\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841434    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-slash\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841462    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-sys\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.842948 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2a29023-9a19-4533-afcf-b3ea0ac28e63-konnectivity-ca\") pod \"konnectivity-agent-vvkr9\" (UID: \"e2a29023-9a19-4533-afcf-b3ea0ac28e63\") " pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c40a55bc-b4ed-4df5-8c3c-a939e7989b12-tmp-dir\") pod \"node-resolver-mz5qt\" (UID: \"c40a55bc-b4ed-4df5-8c3c-a939e7989b12\") " pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841608    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-var-lib-cni-multus\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841637    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-daemon-config\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6zfw\" (UniqueName: \"kubernetes.io/projected/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-kube-api-access-m6zfw\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-run\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-cnibin\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-systemd-units\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841772    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-kubernetes\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841838    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-sys-fs\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.841776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-var-lib-openvswitch\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842149    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-cni-binary-copy\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-sys\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842444    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:10:43.843847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842479    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-sysctl-d\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-device-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c023b43e-30f9-4d56-aa4f-50f5e86724cf-serviceca\") pod \"node-ca-jht5g\" (UID: \"c023b43e-30f9-4d56-aa4f-50f5e86724cf\") " pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-cnibin\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c023b43e-30f9-4d56-aa4f-50f5e86724cf-host\") pod \"node-ca-jht5g\" (UID: \"c023b43e-30f9-4d56-aa4f-50f5e86724cf\") " pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842641    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-system-cni-dir\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842729    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-modprobe-d\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.842823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwjl\" (UniqueName: \"kubernetes.io/projected/c40a55bc-b4ed-4df5-8c3c-a939e7989b12-kube-api-access-tfwjl\") pod \"node-resolver-mz5qt\" (UID: \"c40a55bc-b4ed-4df5-8c3c-a939e7989b12\") " pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.842939    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.843016    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs podName:9cdd97a5-b1fd-4756-9933-1a551767c9c6 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:44.342996942 +0000 UTC m=+2.986966430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs") pod "network-metrics-daemon-6czkd" (UID: "9cdd97a5-b1fd-4756-9933-1a551767c9c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-registration-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843170    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-socket-dir\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-cnibin\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-run\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843210    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-socket-dir-parent\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-systemd\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.844783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843475    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-os-release\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.845697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843494    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-cni-bin\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.845697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-sysctl-conf\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.845697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-systemd\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.845697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843574    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-os-release\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.845697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7cc17ed-e2a2-4226-9287-63c1c56238fc-ovnkube-script-lib\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.845697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds2n7\" (UniqueName: \"kubernetes.io/projected/c7cc17ed-e2a2-4226-9287-63c1c56238fc-kube-api-access-ds2n7\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.845697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-run-netns\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.845697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.843700    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e2a29023-9a19-4533-afcf-b3ea0ac28e63-konnectivity-ca\") pod \"konnectivity-agent-vvkr9\" (UID: \"e2a29023-9a19-4533-afcf-b3ea0ac28e63\") " pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:10:43.846907 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.846880    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/facca3dd-945a-42d5-a32a-b8e4de3f7108-etc-tuned\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.849001 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.848955    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/facca3dd-945a-42d5-a32a-b8e4de3f7108-tmp\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.849109 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.849092    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e2a29023-9a19-4533-afcf-b3ea0ac28e63-agent-certs\") pod \"konnectivity-agent-vvkr9\" (UID: \"e2a29023-9a19-4533-afcf-b3ea0ac28e63\") " pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:10:43.850296 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.850266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qx8\" (UniqueName: \"kubernetes.io/projected/6c0b6e3a-cb60-4b91-be9f-3a50d5472900-kube-api-access-l2qx8\") pod \"aws-ebs-csi-driver-node-25gq5\" (UID: \"6c0b6e3a-cb60-4b91-be9f-3a50d5472900\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:43.851417 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.851393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt4qf\" (UniqueName: \"kubernetes.io/projected/facca3dd-945a-42d5-a32a-b8e4de3f7108-kube-api-access-nt4qf\") pod \"tuned-fk7jp\" (UID: \"facca3dd-945a-42d5-a32a-b8e4de3f7108\") " pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:43.852553 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.852510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw7tb\" (UniqueName: \"kubernetes.io/projected/8b7cd99b-bfab-4d37-a1b4-9d39302ba332-kube-api-access-dw7tb\") pod \"iptables-alerter-5m4kt\" (UID: \"8b7cd99b-bfab-4d37-a1b4-9d39302ba332\") " pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:43.852914 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.852881    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6vww\" (UniqueName: \"kubernetes.io/projected/e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31-kube-api-access-m6vww\") pod \"multus-additional-cni-plugins-j7k5l\" (UID: \"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31\") " pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:43.856200 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.856177    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs8gb\" (UniqueName: \"kubernetes.io/projected/c023b43e-30f9-4d56-aa4f-50f5e86724cf-kube-api-access-hs8gb\") pod \"node-ca-jht5g\" (UID: \"c023b43e-30f9-4d56-aa4f-50f5e86724cf\") " pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:43.858491 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.858465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29nv\" (UniqueName: \"kubernetes.io/projected/9cdd97a5-b1fd-4756-9933-1a551767c9c6-kube-api-access-t29nv\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:43.905219 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.904947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" event={"ID":"eee64e253de37dcdd5d1e7fdc42d07fa","Type":"ContainerStarted","Data":"ab12e5915ee23a5f8424657ab94cd376f14d9de1295c0fda2b37751528706b5d"}
Jul 28 17:10:43.906739 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.906710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" event={"ID":"9646f759e4901fd6d074052a7efe1831","Type":"ContainerStarted","Data":"46c80a0ef58b11537fc388c406eef38935743f2152a2ca8b95efba7c08159d65"}
Jul 28 17:10:43.944002 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.943963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-os-release\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944002 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-run-netns\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-node-log\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-log-socket\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944084    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c40a55bc-b4ed-4df5-8c3c-a939e7989b12-hosts-file\") pod \"node-resolver-mz5qt\" (UID: \"c40a55bc-b4ed-4df5-8c3c-a939e7989b12\") " pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-system-cni-dir\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944161    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-run-openvswitch\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7cc17ed-e2a2-4226-9287-63c1c56238fc-ovn-node-metrics-cert\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-os-release\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-cni-dir\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944250 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-run-k8s-cni-cncf-io\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944287    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-conf-dir\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-hostroot\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944375    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-cni-dir\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-run-multus-certs\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-node-log\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-var-lib-kubelet\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-etc-openvswitch\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-log-socket\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944481    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-run-ovn\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-cni-netd\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944522    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-cni-binary-copy\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-kubelet\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c40a55bc-b4ed-4df5-8c3c-a939e7989b12-hosts-file\") pod \"node-resolver-mz5qt\" (UID: \"c40a55bc-b4ed-4df5-8c3c-a939e7989b12\") " pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-run-systemd\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-kubelet\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.944789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-var-lib-cni-bin\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-run-k8s-cni-cncf-io\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944640    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a0a67e1-6d79-423b-8995-cd624597642f-dbus\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-slash\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-conf-dir\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944708    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-hostroot\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944739    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c40a55bc-b4ed-4df5-8c3c-a939e7989b12-tmp-dir\") pod \"node-resolver-mz5qt\" (UID: \"c40a55bc-b4ed-4df5-8c3c-a939e7989b12\") " pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-run-ovn-kubernetes\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-var-lib-cni-multus\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-daemon-config\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6zfw\" (UniqueName: \"kubernetes.io/projected/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-kube-api-access-m6zfw\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.944842    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-etc-openvswitch\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.944901    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret podName:4a0a67e1-6d79-423b-8995-cd624597642f nodeName:}" failed. No retries permitted until 2026-07-28 17:10:44.444883007 +0000 UTC m=+3.088852478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret") pod "global-pull-secret-syncer-c6vbq" (UID: "4a0a67e1-6d79-423b-8995-cd624597642f") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944949    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-run-multus-certs\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-var-lib-kubelet\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.945712 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4a0a67e1-6d79-423b-8995-cd624597642f-dbus\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-run-ovn\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945020    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-systemd-units\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945051    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-systemd-units\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.944136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-run-netns\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-var-lib-openvswitch\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwjl\" (UniqueName: \"kubernetes.io/projected/c40a55bc-b4ed-4df5-8c3c-a939e7989b12-kube-api-access-tfwjl\") pod \"node-resolver-mz5qt\" (UID: \"c40a55bc-b4ed-4df5-8c3c-a939e7989b12\") " pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945126    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-var-lib-openvswitch\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945132    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-system-cni-dir\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-cnibin\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945166    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-cni-netd\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945041    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-slash\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-socket-dir-parent\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945220    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-run-openvswitch\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945225    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-cni-bin\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7cc17ed-e2a2-4226-9287-63c1c56238fc-ovnkube-script-lib\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-cnibin\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.946588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds2n7\" (UniqueName: \"kubernetes.io/projected/c7cc17ed-e2a2-4226-9287-63c1c56238fc-kube-api-access-ds2n7\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945310    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-run-systemd\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-run-netns\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945353    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-run-netns\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945375    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a0a67e1-6d79-423b-8995-cd624597642f-kubelet-config\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945397    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-socket-dir-parent\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945413    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7cc17ed-e2a2-4226-9287-63c1c56238fc-ovnkube-config\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c40a55bc-b4ed-4df5-8c3c-a939e7989b12-tmp-dir\") pod \"node-resolver-mz5qt\" (UID: \"c40a55bc-b4ed-4df5-8c3c-a939e7989b12\") " pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7cc17ed-e2a2-4226-9287-63c1c56238fc-env-overrides\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c7cc17ed-e2a2-4226-9287-63c1c56238fc-host-cni-bin\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-var-lib-cni-multus\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945522    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4a0a67e1-6d79-423b-8995-cd624597642f-kubelet-config\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-cni-binary-copy\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.945960    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-multus-daemon-config\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.946003    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-etc-kubernetes\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.946064    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-etc-kubernetes\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.946116    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c7cc17ed-e2a2-4226-9287-63c1c56238fc-ovnkube-script-lib\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.946142    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c7cc17ed-e2a2-4226-9287-63c1c56238fc-ovnkube-config\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.947549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.946154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c7cc17ed-e2a2-4226-9287-63c1c56238fc-env-overrides\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.948445 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.946243    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-host-var-lib-cni-bin\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.950157 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.950131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c7cc17ed-e2a2-4226-9287-63c1c56238fc-ovn-node-metrics-cert\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:43.956221 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.956165    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:43.956221 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.956194    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:43.956221 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.956208    2577 projected.go:196] Error preparing data for projected volume kube-api-access-s5lq5 for pod openshift-network-diagnostics/network-check-target-b5btq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:43.956456 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:43.956284    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5 podName:fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:44.456248504 +0000 UTC m=+3.100217977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s5lq5" (UniqueName: "kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5") pod "network-check-target-b5btq" (UID: "fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:43.957448 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.957425    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwjl\" (UniqueName: \"kubernetes.io/projected/c40a55bc-b4ed-4df5-8c3c-a939e7989b12-kube-api-access-tfwjl\") pod \"node-resolver-mz5qt\" (UID: \"c40a55bc-b4ed-4df5-8c3c-a939e7989b12\") " pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:43.958938 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.958900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6zfw\" (UniqueName: \"kubernetes.io/projected/a5d86e2e-21d5-4499-9468-1f98e0d63a2c-kube-api-access-m6zfw\") pod \"multus-9wfsw\" (UID: \"a5d86e2e-21d5-4499-9468-1f98e0d63a2c\") " pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:43.960289 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:43.960262    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds2n7\" (UniqueName: \"kubernetes.io/projected/c7cc17ed-e2a2-4226-9287-63c1c56238fc-kube-api-access-ds2n7\") pod \"ovnkube-node-ck9nr\" (UID: \"c7cc17ed-e2a2-4226-9287-63c1c56238fc\") " pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:44.028484 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.028196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5m4kt"
Jul 28 17:10:44.035566 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.035540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j7k5l"
Jul 28 17:10:44.047785 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.047758    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:10:44.055010 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.054985    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fk7jp"
Jul 28 17:10:44.063751 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.063728    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jht5g"
Jul 28 17:10:44.071903 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.071880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5"
Jul 28 17:10:44.081135 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.081115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:10:44.090073 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.090048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mz5qt"
Jul 28 17:10:44.097247 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.097225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9wfsw"
Jul 28 17:10:44.253606 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.253523    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:44.349621 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.349578    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:44.349785 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.349747    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:44.349833 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.349820    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs podName:9cdd97a5-b1fd-4756-9933-1a551767c9c6 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:45.349801053 +0000 UTC m=+3.993770526 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs") pod "network-metrics-daemon-6czkd" (UID: "9cdd97a5-b1fd-4756-9933-1a551767c9c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:44.450787 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.450743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:44.451000 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.450930    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:44.451063 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.451016    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret podName:4a0a67e1-6d79-423b-8995-cd624597642f nodeName:}" failed. No retries permitted until 2026-07-28 17:10:45.450992422 +0000 UTC m=+4.094961896 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret") pod "global-pull-secret-syncer-c6vbq" (UID: "4a0a67e1-6d79-423b-8995-cd624597642f") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:44.551566 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.551464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:44.551745 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.551683    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:44.551745 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.551712    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:44.551745 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.551727    2577 projected.go:196] Error preparing data for projected volume kube-api-access-s5lq5 for pod openshift-network-diagnostics/network-check-target-b5btq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:44.551909 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.551801    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5 podName:fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:45.551779855 +0000 UTC m=+4.195749338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5lq5" (UniqueName: "kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5") pod "network-check-target-b5btq" (UID: "fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:44.775789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.775749    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:44 +0000 UTC" deadline="2028-01-06 00:39:53.809547027 +0000 UTC"
Jul 28 17:10:44.775789 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.775783    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12631h29m9.033767725s"
Jul 28 17:10:44.893650 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.893579    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:44.899441 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.899409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:44.899607 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:44.899412    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:44.899607 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.899512    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:10:44.899734 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:44.899631    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:10:45.017380 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:45.017346    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a29023_9a19_4533_afcf_b3ea0ac28e63.slice/crio-acae9bbddc8acba83515eb0bb28a1ecb2f62193429aa100fe049db82caaef7fd WatchSource:0}: Error finding container acae9bbddc8acba83515eb0bb28a1ecb2f62193429aa100fe049db82caaef7fd: Status 404 returned error can't find the container with id acae9bbddc8acba83515eb0bb28a1ecb2f62193429aa100fe049db82caaef7fd
Jul 28 17:10:45.035871 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:45.035842    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b7cd99b_bfab_4d37_a1b4_9d39302ba332.slice/crio-a9c99561a2089b2189b5b9dab1346df171c3163720048bc3d67be361d09e2765 WatchSource:0}: Error finding container a9c99561a2089b2189b5b9dab1346df171c3163720048bc3d67be361d09e2765: Status 404 returned error can't find the container with id a9c99561a2089b2189b5b9dab1346df171c3163720048bc3d67be361d09e2765
Jul 28 17:10:45.037053 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:45.037033    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0b6e3a_cb60_4b91_be9f_3a50d5472900.slice/crio-dd5767311aecc9fabbfad46089eb55150f870736069d739307665bf6eb0bf6d7 WatchSource:0}: Error finding container dd5767311aecc9fabbfad46089eb55150f870736069d739307665bf6eb0bf6d7: Status 404 returned error can't find the container with id dd5767311aecc9fabbfad46089eb55150f870736069d739307665bf6eb0bf6d7
Jul 28 17:10:45.038652 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:45.038623    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40a55bc_b4ed_4df5_8c3c_a939e7989b12.slice/crio-33f31f6dcb727d3f63eecb0dbe6380a146951c7b033aafcca444bcbaff565835 WatchSource:0}: Error finding container 33f31f6dcb727d3f63eecb0dbe6380a146951c7b033aafcca444bcbaff565835: Status 404 returned error can't find the container with id 33f31f6dcb727d3f63eecb0dbe6380a146951c7b033aafcca444bcbaff565835
Jul 28 17:10:45.040121 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:45.040052    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cc17ed_e2a2_4226_9287_63c1c56238fc.slice/crio-974c261d123011a763ba68fad109a6e6042e269ec11e2a0a83f4165b5f229bd7 WatchSource:0}: Error finding container 974c261d123011a763ba68fad109a6e6042e269ec11e2a0a83f4165b5f229bd7: Status 404 returned error can't find the container with id 974c261d123011a763ba68fad109a6e6042e269ec11e2a0a83f4165b5f229bd7
Jul 28 17:10:45.040666 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:45.040644    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc023b43e_30f9_4d56_aa4f_50f5e86724cf.slice/crio-dd9643a8900d9034e8caf016a1493ea203904678a48e65904883f14aa8ed500d WatchSource:0}: Error finding container dd9643a8900d9034e8caf016a1493ea203904678a48e65904883f14aa8ed500d: Status 404 returned error can't find the container with id dd9643a8900d9034e8caf016a1493ea203904678a48e65904883f14aa8ed500d
Jul 28 17:10:45.042287 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:45.042264    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d86e2e_21d5_4499_9468_1f98e0d63a2c.slice/crio-6400a203489a70e8dff75c0ed8d34fc7bdd81776d3c457a263d27c18d8271d66 WatchSource:0}: Error finding container 6400a203489a70e8dff75c0ed8d34fc7bdd81776d3c457a263d27c18d8271d66: Status 404 returned error can't find the container with id 6400a203489a70e8dff75c0ed8d34fc7bdd81776d3c457a263d27c18d8271d66
Jul 28 17:10:45.043506 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:45.043483    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacca3dd_945a_42d5_a32a_b8e4de3f7108.slice/crio-9242003e688d26106a322cb38b4b544665a94698d892299ad2b1229961206afc WatchSource:0}: Error finding container 9242003e688d26106a322cb38b4b544665a94698d892299ad2b1229961206afc: Status 404 returned error can't find the container with id 9242003e688d26106a322cb38b4b544665a94698d892299ad2b1229961206afc
Jul 28 17:10:45.045921 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:10:45.045835    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dff0f4_1fbd_46eb_9617_c7e45ee2ab31.slice/crio-2597cf8bc78042097508106dea43ecfb9d28664a95e7cf1e39ba305f85f00e80 WatchSource:0}: Error finding container 2597cf8bc78042097508106dea43ecfb9d28664a95e7cf1e39ba305f85f00e80: Status 404 returned error can't find the container with id 2597cf8bc78042097508106dea43ecfb9d28664a95e7cf1e39ba305f85f00e80
Jul 28 17:10:45.358210 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.357986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:45.358368 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:45.358141    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:45.358368 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:45.358353    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs podName:9cdd97a5-b1fd-4756-9933-1a551767c9c6 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:47.358335309 +0000 UTC m=+6.002304779 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs") pod "network-metrics-daemon-6czkd" (UID: "9cdd97a5-b1fd-4756-9933-1a551767c9c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:45.459197 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.459157    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:45.459393 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:45.459301    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:45.459393 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:45.459373    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret podName:4a0a67e1-6d79-423b-8995-cd624597642f nodeName:}" failed. No retries permitted until 2026-07-28 17:10:47.459354059 +0000 UTC m=+6.103323549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret") pod "global-pull-secret-syncer-c6vbq" (UID: "4a0a67e1-6d79-423b-8995-cd624597642f") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:45.561047 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.560361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:45.561047 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:45.560531    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:45.561047 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:45.560555    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:45.561047 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:45.560568    2577 projected.go:196] Error preparing data for projected volume kube-api-access-s5lq5 for pod openshift-network-diagnostics/network-check-target-b5btq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:45.561047 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:45.560631    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5 podName:fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:47.560611715 +0000 UTC m=+6.204581190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5lq5" (UniqueName: "kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5") pod "network-check-target-b5btq" (UID: "fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:45.776550 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.776501    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:44 +0000 UTC" deadline="2028-03-13 04:20:47.625236083 +0000 UTC"
Jul 28 17:10:45.776550 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.776548    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14243h10m1.848693317s"
Jul 28 17:10:45.904390 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.899805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:45.904390 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:45.899941    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:10:45.918967 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.918931    2577 generic.go:356] "Generic (PLEG): container finished" podID="9646f759e4901fd6d074052a7efe1831" containerID="dd32425c862f33c225e5d0ae78b41d31196a11833a30d8a300fc21446f351076" exitCode=0
Jul 28 17:10:45.919079 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.919031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" event={"ID":"9646f759e4901fd6d074052a7efe1831","Type":"ContainerDied","Data":"dd32425c862f33c225e5d0ae78b41d31196a11833a30d8a300fc21446f351076"}
Jul 28 17:10:45.923514 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.923472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7k5l" event={"ID":"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31","Type":"ContainerStarted","Data":"2597cf8bc78042097508106dea43ecfb9d28664a95e7cf1e39ba305f85f00e80"}
Jul 28 17:10:45.956653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.942178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fk7jp" event={"ID":"facca3dd-945a-42d5-a32a-b8e4de3f7108","Type":"ContainerStarted","Data":"9242003e688d26106a322cb38b4b544665a94698d892299ad2b1229961206afc"}
Jul 28 17:10:45.956653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.947565    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9wfsw" event={"ID":"a5d86e2e-21d5-4499-9468-1f98e0d63a2c","Type":"ContainerStarted","Data":"6400a203489a70e8dff75c0ed8d34fc7bdd81776d3c457a263d27c18d8271d66"}
Jul 28 17:10:45.956653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.950693    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jht5g" event={"ID":"c023b43e-30f9-4d56-aa4f-50f5e86724cf","Type":"ContainerStarted","Data":"dd9643a8900d9034e8caf016a1493ea203904678a48e65904883f14aa8ed500d"}
Jul 28 17:10:45.956653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.953976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" event={"ID":"c7cc17ed-e2a2-4226-9287-63c1c56238fc","Type":"ContainerStarted","Data":"974c261d123011a763ba68fad109a6e6042e269ec11e2a0a83f4165b5f229bd7"}
Jul 28 17:10:45.957313 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.957281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5" event={"ID":"6c0b6e3a-cb60-4b91-be9f-3a50d5472900","Type":"ContainerStarted","Data":"dd5767311aecc9fabbfad46089eb55150f870736069d739307665bf6eb0bf6d7"}
Jul 28 17:10:45.963422 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.962833    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" event={"ID":"eee64e253de37dcdd5d1e7fdc42d07fa","Type":"ContainerStarted","Data":"7e12b87f7590318c7f17af3d83a995dac1923238f4a898cf572dbbaddd6841ed"}
Jul 28 17:10:45.975683 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.975563    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mz5qt" event={"ID":"c40a55bc-b4ed-4df5-8c3c-a939e7989b12","Type":"ContainerStarted","Data":"33f31f6dcb727d3f63eecb0dbe6380a146951c7b033aafcca444bcbaff565835"}
Jul 28 17:10:45.985423 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:45.985391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5m4kt" event={"ID":"8b7cd99b-bfab-4d37-a1b4-9d39302ba332","Type":"ContainerStarted","Data":"a9c99561a2089b2189b5b9dab1346df171c3163720048bc3d67be361d09e2765"}
Jul 28 17:10:46.004465 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:46.004381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvkr9" event={"ID":"e2a29023-9a19-4533-afcf-b3ea0ac28e63","Type":"ContainerStarted","Data":"acae9bbddc8acba83515eb0bb28a1ecb2f62193429aa100fe049db82caaef7fd"}
Jul 28 17:10:46.903341 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:46.899821    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:46.903341 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:46.899969    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:10:46.903341 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:46.900399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:46.903341 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:46.900500    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:10:47.018993 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:47.018901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" event={"ID":"9646f759e4901fd6d074052a7efe1831","Type":"ContainerStarted","Data":"62c0b06ebf7ce6ae38e476905f726ea9b28136a28f38bb27dd3e9bc3b4dee3d7"}
Jul 28 17:10:47.040345 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:47.038978    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-190.ec2.internal" podStartSLOduration=4.038954571 podStartE2EDuration="4.038954571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:45.983556679 +0000 UTC m=+4.627526175" watchObservedRunningTime="2026-07-28 17:10:47.038954571 +0000 UTC m=+5.682924063"
Jul 28 17:10:47.377024 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:47.376925    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:47.377254 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:47.377097    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:47.377254 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:47.377161    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs podName:9cdd97a5-b1fd-4756-9933-1a551767c9c6 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:51.377141407 +0000 UTC m=+10.021110882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs") pod "network-metrics-daemon-6czkd" (UID: "9cdd97a5-b1fd-4756-9933-1a551767c9c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:47.478503 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:47.478463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:47.478692 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:47.478632    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:47.478761 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:47.478693    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret podName:4a0a67e1-6d79-423b-8995-cd624597642f nodeName:}" failed. No retries permitted until 2026-07-28 17:10:51.478672368 +0000 UTC m=+10.122641845 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret") pod "global-pull-secret-syncer-c6vbq" (UID: "4a0a67e1-6d79-423b-8995-cd624597642f") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:47.578847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:47.578780    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:47.579175 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:47.579002    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:47.579175 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:47.579036    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:47.579175 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:47.579050    2577 projected.go:196] Error preparing data for projected volume kube-api-access-s5lq5 for pod openshift-network-diagnostics/network-check-target-b5btq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:47.579175 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:47.579105    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5 podName:fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:51.579086871 +0000 UTC m=+10.223056345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5lq5" (UniqueName: "kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5") pod "network-check-target-b5btq" (UID: "fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:47.904229 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:47.903692    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:47.904229 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:47.903856    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:10:48.899166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:48.899132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:48.899396 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:48.899303    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:10:48.899396 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:48.899337    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:48.899592 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:48.899457    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:10:49.903348 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:49.902833    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:49.903348 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:49.902923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:10:50.900282 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:50.899617    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:50.900282 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:50.899744    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:10:50.900282 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:50.900153    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:50.900282 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:50.900224    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:10:51.414052 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:51.414014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:51.414489 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:51.414229    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:51.414489 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:51.414313    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs podName:9cdd97a5-b1fd-4756-9933-1a551767c9c6 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:59.414293288 +0000 UTC m=+18.058262761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs") pod "network-metrics-daemon-6czkd" (UID: "9cdd97a5-b1fd-4756-9933-1a551767c9c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:51.515387 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:51.515350    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:51.515567 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:51.515484    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:51.515567 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:51.515557    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret podName:4a0a67e1-6d79-423b-8995-cd624597642f nodeName:}" failed. No retries permitted until 2026-07-28 17:10:59.515538207 +0000 UTC m=+18.159507675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret") pod "global-pull-secret-syncer-c6vbq" (UID: "4a0a67e1-6d79-423b-8995-cd624597642f") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:51.616669 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:51.616087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:51.616669 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:51.616261    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:51.616669 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:51.616284    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:51.616669 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:51.616297    2577 projected.go:196] Error preparing data for projected volume kube-api-access-s5lq5 for pod openshift-network-diagnostics/network-check-target-b5btq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:51.616669 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:51.616355    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5 podName:fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:59.616337324 +0000 UTC m=+18.260306816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5lq5" (UniqueName: "kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5") pod "network-check-target-b5btq" (UID: "fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:51.899343 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:51.899261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:51.899504 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:51.899401    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:10:52.899891 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:52.899861    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:52.900358 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:52.899974    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:10:52.900358 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:52.900338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:52.900441 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:52.900425    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:10:53.919263 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:53.918727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:53.919263 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:53.918856    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:10:54.899515 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:54.899483    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:54.899695 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:54.899607    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:10:54.899695 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:54.899654    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:54.899804 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:54.899741    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:10:55.902352 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:55.902323    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:55.902841 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:55.902438    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:10:56.899872 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:56.899835    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:56.900057 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:56.899963    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:10:56.900419 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:56.900385    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:56.900548 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:56.900491    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:10:57.901998 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:57.901961    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:57.902446 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:57.902072    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:10:58.899887 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:58.899852    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:58.900097 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:58.899855    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:58.900097 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:58.899965    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:10:58.900097 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:58.900035    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:10:59.476994 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:59.476953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:10:59.477406 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:59.477094    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:59.477406 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:59.477156    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs podName:9cdd97a5-b1fd-4756-9933-1a551767c9c6 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:15.477137166 +0000 UTC m=+34.121106849 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs") pod "network-metrics-daemon-6czkd" (UID: "9cdd97a5-b1fd-4756-9933-1a551767c9c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:59.577911 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:59.577875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:10:59.578060 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:59.578015    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:59.578104 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:59.578063    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret podName:4a0a67e1-6d79-423b-8995-cd624597642f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:15.578049728 +0000 UTC m=+34.222019197 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret") pod "global-pull-secret-syncer-c6vbq" (UID: "4a0a67e1-6d79-423b-8995-cd624597642f") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:59.678603 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:59.678566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:59.678755 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:59.678736    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:59.678791 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:59.678758    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:59.678791 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:59.678767    2577 projected.go:196] Error preparing data for projected volume kube-api-access-s5lq5 for pod openshift-network-diagnostics/network-check-target-b5btq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:59.678850 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:59.678817    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5 podName:fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:15.678803654 +0000 UTC m=+34.322773122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5lq5" (UniqueName: "kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5") pod "network-check-target-b5btq" (UID: "fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:59.902423 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:10:59.902360    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:10:59.902556 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:10:59.902490    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:00.899370 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:00.899334    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:00.899804 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:00.899472    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:11:00.899804 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:00.899532    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:00.899804 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:00.899658    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:11:01.900049 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:01.900014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:01.900509 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:01.900115    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:02.899303 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:02.899262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:02.899482 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:02.899281    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:02.899482 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:02.899404    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:11:02.899482 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:02.899460    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:11:03.903532 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:03.903164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:03.903532 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:03.903476    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:04.060494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.060290    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31" containerID="55ec54573aebfe2a5ec3ebd40296c088e64ad5c0c6d0b87efbb6eb4a565c730c" exitCode=0
Jul 28 17:11:04.060613 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.060368    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7k5l" event={"ID":"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31","Type":"ContainerDied","Data":"55ec54573aebfe2a5ec3ebd40296c088e64ad5c0c6d0b87efbb6eb4a565c730c"}
Jul 28 17:11:04.062664 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.062634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fk7jp" event={"ID":"facca3dd-945a-42d5-a32a-b8e4de3f7108","Type":"ContainerStarted","Data":"591d74cbf18b89f58153f6b6197178d8a258deca42c8c6a80391954530c4a47d"}
Jul 28 17:11:04.065024 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.064600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9wfsw" event={"ID":"a5d86e2e-21d5-4499-9468-1f98e0d63a2c","Type":"ContainerStarted","Data":"78b00089d7e9b67a3f2f6896cfd1a00fa55f696a3ebe56bf99825d502415dd54"}
Jul 28 17:11:04.066177 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.066157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jht5g" event={"ID":"c023b43e-30f9-4d56-aa4f-50f5e86724cf","Type":"ContainerStarted","Data":"876019e8fa2c8cb1ee423ee01d194667282ce4673afd08c88f436cdd5bc2403c"}
Jul 28 17:11:04.068485 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.068462    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" event={"ID":"c7cc17ed-e2a2-4226-9287-63c1c56238fc","Type":"ContainerStarted","Data":"02a6fd25102b58bb5c8cb9d966076774fd3fb1d4e92fb7609b3906a5cb6bf3d5"}
Jul 28 17:11:04.068570 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.068534    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" event={"ID":"c7cc17ed-e2a2-4226-9287-63c1c56238fc","Type":"ContainerStarted","Data":"a666d6efef6916be13b3fa23cad3a82d7397f41ca42edcdb1b9471339089400f"}
Jul 28 17:11:04.069768 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.069746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5" event={"ID":"6c0b6e3a-cb60-4b91-be9f-3a50d5472900","Type":"ContainerStarted","Data":"61eb57fc61339909d6007fa58f46ea1f513878780a1a13565813c75e87613717"}
Jul 28 17:11:04.070932 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.070912    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mz5qt" event={"ID":"c40a55bc-b4ed-4df5-8c3c-a939e7989b12","Type":"ContainerStarted","Data":"d6b14ca34575c12b2606fd3bb6535a3e8afd88854ded3af963a2dec484fc92e6"}
Jul 28 17:11:04.072454 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.072434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vvkr9" event={"ID":"e2a29023-9a19-4533-afcf-b3ea0ac28e63","Type":"ContainerStarted","Data":"0238ec54f3a7b5a310f3210c6fabc967180220c403a78a6d0cc12399aae58a30"}
Jul 28 17:11:04.090694 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.090637    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-190.ec2.internal" podStartSLOduration=21.090620827 podStartE2EDuration="21.090620827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:47.040229064 +0000 UTC m=+5.684198537" watchObservedRunningTime="2026-07-28 17:11:04.090620827 +0000 UTC m=+22.734590317"
Jul 28 17:11:04.103693 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.103634    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vvkr9" podStartSLOduration=11.111322297 podStartE2EDuration="22.103617657s" totalImagesPullingTime="10.99229536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:04.103227364 +0000 UTC m=+22.747196855" watchObservedRunningTime="2026-07-28 17:11:04.103617657 +0000 UTC m=+22.747587148"
Jul 28 17:11:04.135000 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.134960    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fk7jp" podStartSLOduration=3.386407191 podStartE2EDuration="22.134945253s" totalImagesPullingTime="18.748538062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:04.119632308 +0000 UTC m=+22.763601796" watchObservedRunningTime="2026-07-28 17:11:04.134945253 +0000 UTC m=+22.778914743"
Jul 28 17:11:04.135328 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.135296    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mz5qt" podStartSLOduration=3.412094229 podStartE2EDuration="22.135286281s" totalImagesPullingTime="18.723192052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:04.134656175 +0000 UTC m=+22.778625690" watchObservedRunningTime="2026-07-28 17:11:04.135286281 +0000 UTC m=+22.779255772"
Jul 28 17:11:04.149838 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.149802    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jht5g" podStartSLOduration=3.42936542 podStartE2EDuration="22.149785215s" totalImagesPullingTime="18.720419795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:04.149687303 +0000 UTC m=+22.793656785" watchObservedRunningTime="2026-07-28 17:11:04.149785215 +0000 UTC m=+22.793754704"
Jul 28 17:11:04.168929 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.168887    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9wfsw" podStartSLOduration=3.398283122 podStartE2EDuration="22.168867215s" totalImagesPullingTime="18.770584093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:04.168770214 +0000 UTC m=+22.812739704" watchObservedRunningTime="2026-07-28 17:11:04.168867215 +0000 UTC m=+22.812836707"
Jul 28 17:11:04.899747 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.899718    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:04.899920 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:04.899720    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:04.899920 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:04.899850    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:11:04.900022 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:04.900005    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:11:05.079135 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:05.078331    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" event={"ID":"c7cc17ed-e2a2-4226-9287-63c1c56238fc","Type":"ContainerStarted","Data":"0e4240d1d19621069dd23f3f93940a88d7e312317e7f81523b50b84d8c830fe3"}
Jul 28 17:11:05.079135 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:05.078403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" event={"ID":"c7cc17ed-e2a2-4226-9287-63c1c56238fc","Type":"ContainerStarted","Data":"b487a09a236daad42d287d2ed7fb8e0c03766a799bd2c074cbee1277db375f7a"}
Jul 28 17:11:05.079135 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:05.078434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" event={"ID":"c7cc17ed-e2a2-4226-9287-63c1c56238fc","Type":"ContainerStarted","Data":"2beaceb0a8eec124ff604dd12abe89024043b6d339523b84052140a02e6060b2"}
Jul 28 17:11:05.079135 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:05.078465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" event={"ID":"c7cc17ed-e2a2-4226-9287-63c1c56238fc","Type":"ContainerStarted","Data":"85524151e2fd0523ea4eb05e55fb4310337500f5a20d67b0ac2024e7c4d298f6"}
Jul 28 17:11:05.357776 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:05.357752    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:11:05.796650 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:05.796516    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:11:05.357773699Z","UUID":"7b09996a-7052-4b6b-996d-1a0efa5c0291","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:11:05.798438 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:05.798413    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:11:05.798564 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:05.798450    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:11:05.900302 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:05.899175    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:05.900302 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:05.899306    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:06.082743 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:06.082674    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5m4kt" event={"ID":"8b7cd99b-bfab-4d37-a1b4-9d39302ba332","Type":"ContainerStarted","Data":"c3bbe8b6540d0b39b47a43d49f9137f65eff736cf5d797085f3fbbde0350d6b3"}
Jul 28 17:11:06.084999 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:06.084973    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5" event={"ID":"6c0b6e3a-cb60-4b91-be9f-3a50d5472900","Type":"ContainerStarted","Data":"b6742345ee50a47ce54ab41c489e0444a3a88ddbcf595d3a16554409df8b1913"}
Jul 28 17:11:06.100794 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:06.100742    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5m4kt" podStartSLOduration=5.374180008 podStartE2EDuration="24.100724682s" totalImagesPullingTime="18.726544674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:06.099642202 +0000 UTC m=+24.743611692" watchObservedRunningTime="2026-07-28 17:11:06.100724682 +0000 UTC m=+24.744694177"
Jul 28 17:11:06.899471 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:06.899438    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:06.899637 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:06.899439    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:06.899637 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:06.899562    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:11:06.899758 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:06.899663    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:11:07.092126 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:07.091871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" event={"ID":"c7cc17ed-e2a2-4226-9287-63c1c56238fc","Type":"ContainerStarted","Data":"771d20f5d86e9077e0411d129b6e5385f23ed7b5e3306623df8d62c4253fa8c1"}
Jul 28 17:11:07.094186 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:07.094157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5" event={"ID":"6c0b6e3a-cb60-4b91-be9f-3a50d5472900","Type":"ContainerStarted","Data":"e7e8f60eee2cf323cac902f388dd15e0d40248119a2d07f20fac9d4f1dc4f338"}
Jul 28 17:11:07.117132 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:07.117084    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-25gq5" podStartSLOduration=4.151054985 podStartE2EDuration="25.117066114s" totalImagesPullingTime="20.966011129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:07.116502625 +0000 UTC m=+25.760472116" watchObservedRunningTime="2026-07-28 17:11:07.117066114 +0000 UTC m=+25.761035606"
Jul 28 17:11:07.902629 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:07.902602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:07.902817 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:07.902710    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:08.394117 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:08.394094    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:11:08.394598 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:08.394582    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:11:08.899849 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:08.899782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:08.899975 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:08.899787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:08.899975 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:08.899889    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:11:08.899975 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:08.899960    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:11:09.102747 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:09.102719    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31" containerID="26c818d9ca6a1e00d475a1fcf429b41836accd1020dd36f6a717220ba92aa8ac" exitCode=0
Jul 28 17:11:09.102923 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:09.102793    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7k5l" event={"ID":"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31","Type":"ContainerDied","Data":"26c818d9ca6a1e00d475a1fcf429b41836accd1020dd36f6a717220ba92aa8ac"}
Jul 28 17:11:09.106288 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:09.106261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" event={"ID":"c7cc17ed-e2a2-4226-9287-63c1c56238fc","Type":"ContainerStarted","Data":"944ff086690474aec237d51dc08bbe646324a4743a46b5ec5f54bfab311106d0"}
Jul 28 17:11:09.106467 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:09.106443    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:11:09.106908 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:09.106892    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vvkr9"
Jul 28 17:11:09.173664 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:09.173590    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr" podStartSLOduration=8.384419086 podStartE2EDuration="27.173575599s" totalImagesPullingTime="18.789156513s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:09.173264285 +0000 UTC m=+27.817233775" watchObservedRunningTime="2026-07-28 17:11:09.173575599 +0000 UTC m=+27.817545089"
Jul 28 17:11:09.899113 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:09.899082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:09.899837 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:09.899193    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:10.108452 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.108385    2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 28 17:11:10.108910 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.108888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:11:10.109038 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.108966    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:11:10.123572 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.123550    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:11:10.123816 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.123801    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:11:10.645080 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.645002    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:11:10.646649 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.646615    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b5btq"]
Jul 28 17:11:10.646779 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.646723    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:10.646842 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:10.646801    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:10.649048 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.649029    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c6vbq"]
Jul 28 17:11:10.649147 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.649109    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:10.649188 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:10.649173    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:11:10.660528 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.660509    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6czkd"]
Jul 28 17:11:10.660628 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:10.660601    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:10.660722 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:10.660702    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:11:11.112590 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:11.112515    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31" containerID="306002dcbf1fd04efbf098ba9ef31cdf3518dab9798118b47e00059b31dcfdcb" exitCode=0
Jul 28 17:11:11.113010 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:11.112595    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7k5l" event={"ID":"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31","Type":"ContainerDied","Data":"306002dcbf1fd04efbf098ba9ef31cdf3518dab9798118b47e00059b31dcfdcb"}
Jul 28 17:11:11.900152 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:11.899989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:11.900348 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:11.900056    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:11.900348 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:11.900204    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:11:11.900348 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:11.900291    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:12.899536 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:12.899511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:12.900078 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:12.899626    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:11:13.118834 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:13.118796    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31" containerID="b69b053260d232a5dead561cce6478493ee9385b537c199c9a73cfb78e1b3992" exitCode=0
Jul 28 17:11:13.118971 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:13.118857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7k5l" event={"ID":"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31","Type":"ContainerDied","Data":"b69b053260d232a5dead561cce6478493ee9385b537c199c9a73cfb78e1b3992"}
Jul 28 17:11:13.899858 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:13.899824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:13.900564 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:13.899824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:13.900564 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:13.899993    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:11:13.900564 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:13.900110    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:14.899943 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:14.899909    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:14.900397 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:14.900057    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6vbq" podUID="4a0a67e1-6d79-423b-8995-cd624597642f"
Jul 28 17:11:15.502401 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:15.502372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:15.502554 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.502518    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:11:15.502604 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.502580    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs podName:9cdd97a5-b1fd-4756-9933-1a551767c9c6 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:47.502565102 +0000 UTC m=+66.146534577 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs") pod "network-metrics-daemon-6czkd" (UID: "9cdd97a5-b1fd-4756-9933-1a551767c9c6") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:11:15.603289 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:15.603260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:15.603435 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.603394    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:11:15.603478 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.603449    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret podName:4a0a67e1-6d79-423b-8995-cd624597642f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:47.603435548 +0000 UTC m=+66.247405017 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret") pod "global-pull-secret-syncer-c6vbq" (UID: "4a0a67e1-6d79-423b-8995-cd624597642f") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:11:15.704502 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:15.704471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:15.704666 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.704644    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:11:15.704666 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.704666    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:11:15.704739 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.704676    2577 projected.go:196] Error preparing data for projected volume kube-api-access-s5lq5 for pod openshift-network-diagnostics/network-check-target-b5btq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:11:15.704739 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.704728    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5 podName:fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:47.704713485 +0000 UTC m=+66.348682952 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s5lq5" (UniqueName: "kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5") pod "network-check-target-b5btq" (UID: "fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:11:15.903479 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:15.903449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:15.903878 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.903577    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6czkd" podUID="9cdd97a5-b1fd-4756-9933-1a551767c9c6"
Jul 28 17:11:15.903878 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:15.903594    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:15.903878 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:15.903696    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b5btq" podUID="fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3"
Jul 28 17:11:16.660397 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.660371    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-190.ec2.internal" event="NodeReady"
Jul 28 17:11:16.660552 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.660507    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:11:16.703437 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.703406    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d7rfz"]
Jul 28 17:11:16.734640 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.734609    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wmsx9"]
Jul 28 17:11:16.734813 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.734765    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:16.737603 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.737435    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:11:16.737603 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.737527    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-947wh\""
Jul 28 17:11:16.737603 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.737527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:11:16.750511 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.750490    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wmsx9"]
Jul 28 17:11:16.750628 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.750552    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7rfz"]
Jul 28 17:11:16.750683 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.750673    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wmsx9"
Jul 28 17:11:16.755190 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.754859    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:11:16.755314 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.755235    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:11:16.755531 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.755346    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:11:16.755996 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.755842    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-sx2rm\""
Jul 28 17:11:16.898976 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.898945    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:16.901990 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.901969    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:11:16.912452 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.912404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c648388c-85fe-4368-b838-b3fb9f356628-tmp-dir\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:16.912452 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.912444    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c648388c-85fe-4368-b838-b3fb9f356628-config-volume\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:16.912893 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.912475    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c-cert\") pod \"ingress-canary-wmsx9\" (UID: \"01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c\") " pod="openshift-ingress-canary/ingress-canary-wmsx9"
Jul 28 17:11:16.912893 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.912559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:16.912893 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.912590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5468f\" (UniqueName: \"kubernetes.io/projected/01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c-kube-api-access-5468f\") pod \"ingress-canary-wmsx9\" (UID: \"01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c\") " pod="openshift-ingress-canary/ingress-canary-wmsx9"
Jul 28 17:11:16.912893 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:16.912610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k7hb\" (UniqueName: \"kubernetes.io/projected/c648388c-85fe-4368-b838-b3fb9f356628-kube-api-access-4k7hb\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:17.013692 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.013653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k7hb\" (UniqueName: \"kubernetes.io/projected/c648388c-85fe-4368-b838-b3fb9f356628-kube-api-access-4k7hb\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:17.013885 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.013834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c648388c-85fe-4368-b838-b3fb9f356628-tmp-dir\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:17.013885 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.013866    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c648388c-85fe-4368-b838-b3fb9f356628-config-volume\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:17.013983 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.013895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c-cert\") pod \"ingress-canary-wmsx9\" (UID: \"01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c\") " pod="openshift-ingress-canary/ingress-canary-wmsx9"
Jul 28 17:11:17.013983 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.013940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:17.013983 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.013966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5468f\" (UniqueName: \"kubernetes.io/projected/01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c-kube-api-access-5468f\") pod \"ingress-canary-wmsx9\" (UID: \"01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c\") " pod="openshift-ingress-canary/ingress-canary-wmsx9"
Jul 28 17:11:17.014349 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:17.014330    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:17.014427 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:17.014395    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls podName:c648388c-85fe-4368-b838-b3fb9f356628 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:17.514375833 +0000 UTC m=+36.158345301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls") pod "dns-default-d7rfz" (UID: "c648388c-85fe-4368-b838-b3fb9f356628") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:17.014592 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.014234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c648388c-85fe-4368-b838-b3fb9f356628-tmp-dir\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:17.014798 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.014778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c648388c-85fe-4368-b838-b3fb9f356628-config-volume\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:17.018669 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.018642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c-cert\") pod \"ingress-canary-wmsx9\" (UID: \"01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c\") " pod="openshift-ingress-canary/ingress-canary-wmsx9"
Jul 28 17:11:17.025626 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.025599    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5468f\" (UniqueName: \"kubernetes.io/projected/01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c-kube-api-access-5468f\") pod \"ingress-canary-wmsx9\" (UID: \"01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c\") " pod="openshift-ingress-canary/ingress-canary-wmsx9"
Jul 28 17:11:17.025840 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.025820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k7hb\" (UniqueName: \"kubernetes.io/projected/c648388c-85fe-4368-b838-b3fb9f356628-kube-api-access-4k7hb\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:17.060372 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.060340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wmsx9"
Jul 28 17:11:17.298954 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.298782    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wmsx9"]
Jul 28 17:11:17.344114 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:11:17.344084    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01cc0f78_cb16_4263_8ed2_0cb78bb0ef3c.slice/crio-b38eaaddfd514377e2b4baa4c0842bbe1acaf667cfd1c21b972cac2fd017195f WatchSource:0}: Error finding container b38eaaddfd514377e2b4baa4c0842bbe1acaf667cfd1c21b972cac2fd017195f: Status 404 returned error can't find the container with id b38eaaddfd514377e2b4baa4c0842bbe1acaf667cfd1c21b972cac2fd017195f
Jul 28 17:11:17.518072 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.518033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:17.518213 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:17.518194    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:17.518286 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:17.518276    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls podName:c648388c-85fe-4368-b838-b3fb9f356628 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:18.518260943 +0000 UTC m=+37.162230424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls") pod "dns-default-d7rfz" (UID: "c648388c-85fe-4368-b838-b3fb9f356628") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:17.899338 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.899307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:17.899696 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.899497    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:17.903537 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.903466    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:11:17.903537 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.903473    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:11:17.903537 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.903482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hxs5k\""
Jul 28 17:11:17.903537 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.903482    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sw5vg\""
Jul 28 17:11:17.903537 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:17.903536    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:11:18.130837 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:18.130802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wmsx9" event={"ID":"01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c","Type":"ContainerStarted","Data":"b38eaaddfd514377e2b4baa4c0842bbe1acaf667cfd1c21b972cac2fd017195f"}
Jul 28 17:11:18.133924 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:18.133891    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31" containerID="1d028740c0af724d222e6b8ba4bc468cd5df4373a07e119faceed48fe858f213" exitCode=0
Jul 28 17:11:18.134048 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:18.133939    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7k5l" event={"ID":"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31","Type":"ContainerDied","Data":"1d028740c0af724d222e6b8ba4bc468cd5df4373a07e119faceed48fe858f213"}
Jul 28 17:11:18.527300 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:18.527197    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:18.527447 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:18.527352    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:18.527447 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:18.527438    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls podName:c648388c-85fe-4368-b838-b3fb9f356628 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:20.527414895 +0000 UTC m=+39.171384372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls") pod "dns-default-d7rfz" (UID: "c648388c-85fe-4368-b838-b3fb9f356628") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:19.139983 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:19.139951    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31" containerID="1051fa51f27fb46719aba1a0f56d8517719df2ac1ae8b67ef85f9d624e54d003" exitCode=0
Jul 28 17:11:19.140408 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:19.139995    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7k5l" event={"ID":"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31","Type":"ContainerDied","Data":"1051fa51f27fb46719aba1a0f56d8517719df2ac1ae8b67ef85f9d624e54d003"}
Jul 28 17:11:20.143588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:20.143429    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wmsx9" event={"ID":"01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c","Type":"ContainerStarted","Data":"c20e064e320c5d42c9bed43f70e586d523bae6aa1944dc3f3367e4ff63c0cd5a"}
Jul 28 17:11:20.146406 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:20.146381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j7k5l" event={"ID":"e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31","Type":"ContainerStarted","Data":"35f1156ad529c2246ba02a62dfe9459dc9e11ae9921150d937edd6b7d6d67d8d"}
Jul 28 17:11:20.162320 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:20.162282    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wmsx9" podStartSLOduration=1.806437949 podStartE2EDuration="4.162268266s" totalImagesPullingTime="2.355830317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:20.160866284 +0000 UTC m=+38.804835774" watchObservedRunningTime="2026-07-28 17:11:20.162268266 +0000 UTC m=+38.806237756"
Jul 28 17:11:20.186671 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:20.186632    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j7k5l" podStartSLOduration=8.816879735 podStartE2EDuration="38.186620227s" totalImagesPullingTime="29.369740492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:20.184835391 +0000 UTC m=+38.828804882" watchObservedRunningTime="2026-07-28 17:11:20.186620227 +0000 UTC m=+38.830589717"
Jul 28 17:11:20.540782 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:20.540677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:20.540919 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:20.540818    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:20.540919 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:20.540879    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls podName:c648388c-85fe-4368-b838-b3fb9f356628 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:24.540865345 +0000 UTC m=+43.184834812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls") pod "dns-default-d7rfz" (UID: "c648388c-85fe-4368-b838-b3fb9f356628") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:24.562396 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:24.562360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:24.562874 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:24.562465    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:24.562874 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:24.562514    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls podName:c648388c-85fe-4368-b838-b3fb9f356628 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:32.562501767 +0000 UTC m=+51.206471236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls") pod "dns-default-d7rfz" (UID: "c648388c-85fe-4368-b838-b3fb9f356628") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:32.620707 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:32.620672    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:32.621139 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:32.620817    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:32.621139 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:32.620891    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls podName:c648388c-85fe-4368-b838-b3fb9f356628 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:48.620871686 +0000 UTC m=+67.264841171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls") pod "dns-default-d7rfz" (UID: "c648388c-85fe-4368-b838-b3fb9f356628") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:42.124846 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:42.124819    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ck9nr"
Jul 28 17:11:47.526196 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.526152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:11:47.534610 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.534586    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:11:47.536666 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:47.536652    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:11:47.536718 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:47.536708    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs podName:9cdd97a5-b1fd-4756-9933-1a551767c9c6 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:51.536691708 +0000 UTC m=+130.180661176 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs") pod "network-metrics-daemon-6czkd" (UID: "9cdd97a5-b1fd-4756-9933-1a551767c9c6") : secret "metrics-daemon-secret" not found
Jul 28 17:11:47.626542 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.626509    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:47.629679 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.629663    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:11:47.640932 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.640902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4a0a67e1-6d79-423b-8995-cd624597642f-original-pull-secret\") pod \"global-pull-secret-syncer-c6vbq\" (UID: \"4a0a67e1-6d79-423b-8995-cd624597642f\") " pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:47.727693 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.727645    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:47.730945 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.730929    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:11:47.741774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.741753    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:11:47.751690 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.751670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5lq5\" (UniqueName: \"kubernetes.io/projected/fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3-kube-api-access-s5lq5\") pod \"network-check-target-b5btq\" (UID: \"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3\") " pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:47.809892 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.809823    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6vbq"
Jul 28 17:11:47.913221 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.913198    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hxs5k\""
Jul 28 17:11:47.920966 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.920937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:47.953007 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:47.952978    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c6vbq"]
Jul 28 17:11:48.049581 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:48.049561    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b5btq"]
Jul 28 17:11:48.051701 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:11:48.051683    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd1d9d5f_6d34_4b5c_909a_56b7532fc4b3.slice/crio-18a5774af636f2034f77873fb2df88beb943757dfd8913e2ea20c5bcd1b60f73 WatchSource:0}: Error finding container 18a5774af636f2034f77873fb2df88beb943757dfd8913e2ea20c5bcd1b60f73: Status 404 returned error can't find the container with id 18a5774af636f2034f77873fb2df88beb943757dfd8913e2ea20c5bcd1b60f73
Jul 28 17:11:48.207530 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:48.207493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c6vbq" event={"ID":"4a0a67e1-6d79-423b-8995-cd624597642f","Type":"ContainerStarted","Data":"ab46a06224dc18f2c8f56cf204602ed58f0fbbd43d34c7bfa6e1f6cdbf19f85f"}
Jul 28 17:11:48.208509 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:48.208482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b5btq" event={"ID":"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3","Type":"ContainerStarted","Data":"18a5774af636f2034f77873fb2df88beb943757dfd8913e2ea20c5bcd1b60f73"}
Jul 28 17:11:48.633930 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:48.633466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:11:48.633930 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:48.633611    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:48.633930 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:48.633688    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls podName:c648388c-85fe-4368-b838-b3fb9f356628 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:20.633663371 +0000 UTC m=+99.277632843 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls") pod "dns-default-d7rfz" (UID: "c648388c-85fe-4368-b838-b3fb9f356628") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:52.032610 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.032574    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj"]
Jul 28 17:11:52.056030 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.055996    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-vjx7p"]
Jul 28 17:11:52.056227 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.056160    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj"
Jul 28 17:11:52.059429 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.059403    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:52.059429 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.059428    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-l86hf\""
Jul 28 17:11:52.059594 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.059438    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 17:11:52.072135 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.072108    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-vjx7p"]
Jul 28 17:11:52.072135 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.072139    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj"]
Jul 28 17:11:52.072321 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.072230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.076831 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.076664    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wnk6z\""
Jul 28 17:11:52.076831 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.076699    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 17:11:52.076831 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.076824    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:52.077046 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.077027    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 17:11:52.077276 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.077234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 17:11:52.082413 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.082394    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 17:11:52.141507 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.141475    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64cc6c558f-tgnb7"]
Jul 28 17:11:52.158381 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.158349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a54fd9e3-e867-475a-ad77-09acb484c61e-config\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.158550 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.158386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wbvv\" (UniqueName: \"kubernetes.io/projected/a54fd9e3-e867-475a-ad77-09acb484c61e-kube-api-access-6wbvv\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.158550 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.158413    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a54fd9e3-e867-475a-ad77-09acb484c61e-serving-cert\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.158682 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.158548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a54fd9e3-e867-475a-ad77-09acb484c61e-trusted-ca\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.158682 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.158615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhjf\" (UniqueName: \"kubernetes.io/projected/a1a7a1d5-1f22-42b4-8776-a7a71eee2200-kube-api-access-vbhjf\") pod \"volume-data-source-validator-549c6d54cf-7ktzj\" (UID: \"a1a7a1d5-1f22-42b4-8776-a7a71eee2200\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj"
Jul 28 17:11:52.160156 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.160131    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.172588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.172551    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 17:11:52.172706 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.172616    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 17:11:52.174036 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.173852    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 17:11:52.176896 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.176865    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 17:11:52.177001 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.176942    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 17:11:52.177498 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.177399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-w8t7b\""
Jul 28 17:11:52.181748 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.181728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64cc6c558f-tgnb7"]
Jul 28 17:11:52.192222 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.192196    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 17:11:52.259825 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.259786    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhjf\" (UniqueName: \"kubernetes.io/projected/a1a7a1d5-1f22-42b4-8776-a7a71eee2200-kube-api-access-vbhjf\") pod \"volume-data-source-validator-549c6d54cf-7ktzj\" (UID: \"a1a7a1d5-1f22-42b4-8776-a7a71eee2200\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj"
Jul 28 17:11:52.259825 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.259824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a54fd9e3-e867-475a-ad77-09acb484c61e-config\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.260032 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.259841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wbvv\" (UniqueName: \"kubernetes.io/projected/a54fd9e3-e867-475a-ad77-09acb484c61e-kube-api-access-6wbvv\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.260032 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.259863    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a54fd9e3-e867-475a-ad77-09acb484c61e-serving-cert\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.260032 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.259891    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-default-certificate\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.260190 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.260023    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.260190 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.260075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.260190 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.260162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rdh\" (UniqueName: \"kubernetes.io/projected/eb22b701-7530-4ee7-9116-29d6dfde5b0a-kube-api-access-n5rdh\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.260327 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.260279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-stats-auth\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.260381 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.260337    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a54fd9e3-e867-475a-ad77-09acb484c61e-trusted-ca\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.260665 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.260644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a54fd9e3-e867-475a-ad77-09acb484c61e-config\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.261194 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.261171    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a54fd9e3-e867-475a-ad77-09acb484c61e-trusted-ca\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.262634 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.262607    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a54fd9e3-e867-475a-ad77-09acb484c61e-serving-cert\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.280223 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.280190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhjf\" (UniqueName: \"kubernetes.io/projected/a1a7a1d5-1f22-42b4-8776-a7a71eee2200-kube-api-access-vbhjf\") pod \"volume-data-source-validator-549c6d54cf-7ktzj\" (UID: \"a1a7a1d5-1f22-42b4-8776-a7a71eee2200\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj"
Jul 28 17:11:52.280975 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.280952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wbvv\" (UniqueName: \"kubernetes.io/projected/a54fd9e3-e867-475a-ad77-09acb484c61e-kube-api-access-6wbvv\") pod \"console-operator-584dcb6b54-vjx7p\" (UID: \"a54fd9e3-e867-475a-ad77-09acb484c61e\") " pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.361102 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.361038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-default-certificate\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.361102 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.361069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.361102 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.361085    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.361331 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.361113    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rdh\" (UniqueName: \"kubernetes.io/projected/eb22b701-7530-4ee7-9116-29d6dfde5b0a-kube-api-access-n5rdh\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.361331 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.361156    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-stats-auth\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.361331 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:52.361224    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:52.361331 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:52.361269    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:52.861226415 +0000 UTC m=+71.505195901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:52.361331 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:52.361308    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:52.861291474 +0000 UTC m=+71.505260953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : secret "router-metrics-certs-default" not found
Jul 28 17:11:52.363374 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.363347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-default-certificate\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.363501 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.363482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-stats-auth\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.367295 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.367277    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj"
Jul 28 17:11:52.375236 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.375213    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rdh\" (UniqueName: \"kubernetes.io/projected/eb22b701-7530-4ee7-9116-29d6dfde5b0a-kube-api-access-n5rdh\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.382064 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.382047    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:11:52.847839 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.847809    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-vjx7p"]
Jul 28 17:11:52.850457 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:11:52.850432    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54fd9e3_e867_475a_ad77_09acb484c61e.slice/crio-5b39ebc124bfd368846d754b0e21811113b701467dde54cc81b421c49a6f79ce WatchSource:0}: Error finding container 5b39ebc124bfd368846d754b0e21811113b701467dde54cc81b421c49a6f79ce: Status 404 returned error can't find the container with id 5b39ebc124bfd368846d754b0e21811113b701467dde54cc81b421c49a6f79ce
Jul 28 17:11:52.859830 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.859812    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj"]
Jul 28 17:11:52.861643 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:11:52.861621    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a7a1d5_1f22_42b4_8776_a7a71eee2200.slice/crio-bf6344a5a28b485b4e5b5b5513c6519646804784041b50bc9d6a56591549c967 WatchSource:0}: Error finding container bf6344a5a28b485b4e5b5b5513c6519646804784041b50bc9d6a56591549c967: Status 404 returned error can't find the container with id bf6344a5a28b485b4e5b5b5513c6519646804784041b50bc9d6a56591549c967
Jul 28 17:11:52.863215 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.863191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.863427 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:52.863223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:52.863427 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:52.863331    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:52.863427 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:52.863366    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:53.863347019 +0000 UTC m=+72.507316499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:52.863427 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:52.863387    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:53.863378071 +0000 UTC m=+72.507347539 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : secret "router-metrics-certs-default" not found
Jul 28 17:11:53.224778 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:53.224746    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c6vbq" event={"ID":"4a0a67e1-6d79-423b-8995-cd624597642f","Type":"ContainerStarted","Data":"a360f7cb8e0ef1067ff43317baa3b325b1a781fa69965b6932a77c4503355eac"}
Jul 28 17:11:53.226272 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:53.226232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b5btq" event={"ID":"fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3","Type":"ContainerStarted","Data":"6f392602b686bb6695e475b5340b60c2500cea0a31d5cf6a39346663d66d849a"}
Jul 28 17:11:53.226383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:53.226308    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:11:53.227290 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:53.227269    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj" event={"ID":"a1a7a1d5-1f22-42b4-8776-a7a71eee2200","Type":"ContainerStarted","Data":"bf6344a5a28b485b4e5b5b5513c6519646804784041b50bc9d6a56591549c967"}
Jul 28 17:11:53.228108 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:53.228090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p" event={"ID":"a54fd9e3-e867-475a-ad77-09acb484c61e","Type":"ContainerStarted","Data":"5b39ebc124bfd368846d754b0e21811113b701467dde54cc81b421c49a6f79ce"}
Jul 28 17:11:53.246058 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:53.246023    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-c6vbq" podStartSLOduration=66.484386308 podStartE2EDuration="1m11.246010433s" totalImagesPullingTime="4.761624125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:53.244776764 +0000 UTC m=+71.888746315" watchObservedRunningTime="2026-07-28 17:11:53.246010433 +0000 UTC m=+71.889979901"
Jul 28 17:11:53.264285 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:53.264201    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b5btq" podStartSLOduration=66.608084063 podStartE2EDuration="1m11.264188775s" totalImagesPullingTime="4.656104712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:53.263606036 +0000 UTC m=+71.907575544" watchObservedRunningTime="2026-07-28 17:11:53.264188775 +0000 UTC m=+71.908158267"
Jul 28 17:11:53.870431 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:53.870388    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:53.870431 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:53.870441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:53.870696 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:53.870562    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:55.870541985 +0000 UTC m=+74.514511458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:53.870696 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:53.870598    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:53.870696 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:53.870665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:55.870648521 +0000 UTC m=+74.514618002 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : secret "router-metrics-certs-default" not found
Jul 28 17:11:55.234862 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:55.234825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj" event={"ID":"a1a7a1d5-1f22-42b4-8776-a7a71eee2200","Type":"ContainerStarted","Data":"5b48875922bbdc2b7cb9fed14e11711e721fd4c70795ef0e8b641f75a56f5498"}
Jul 28 17:11:55.236441 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:55.236424    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/0.log"
Jul 28 17:11:55.236543 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:55.236458    2577 generic.go:356] "Generic (PLEG): container finished" podID="a54fd9e3-e867-475a-ad77-09acb484c61e" containerID="c84df7475f3c58d13edb8d15810c462c85827733167aabc50b48a75c732adad8" exitCode=255
Jul 28 17:11:55.236543 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:55.236503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p" event={"ID":"a54fd9e3-e867-475a-ad77-09acb484c61e","Type":"ContainerDied","Data":"c84df7475f3c58d13edb8d15810c462c85827733167aabc50b48a75c732adad8"}
Jul 28 17:11:55.236703 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:55.236691    2577 scope.go:117] "RemoveContainer" containerID="c84df7475f3c58d13edb8d15810c462c85827733167aabc50b48a75c732adad8"
Jul 28 17:11:55.255025 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:55.254985    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7ktzj" podStartSLOduration=2.352212943 podStartE2EDuration="4.254971756s" totalImagesPullingTime="1.902758813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:55.254037279 +0000 UTC m=+73.898006770" watchObservedRunningTime="2026-07-28 17:11:55.254971756 +0000 UTC m=+73.898941246"
Jul 28 17:11:55.889043 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:55.889005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:55.889043 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:55.889052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:55.889354 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:55.889172    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:55.889354 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:55.889225    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:59.88920814 +0000 UTC m=+78.533177607 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : secret "router-metrics-certs-default" not found
Jul 28 17:11:55.889354 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:55.889293    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:59.889283546 +0000 UTC m=+78.533253014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:56.241083 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:56.241011    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:11:56.241505 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:56.241439    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/0.log"
Jul 28 17:11:56.241505 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:56.241469    2577 generic.go:356] "Generic (PLEG): container finished" podID="a54fd9e3-e867-475a-ad77-09acb484c61e" containerID="00d577dd9a77787edc2b669c2b394dc8968cc91892410b172b7870758703213c" exitCode=255
Jul 28 17:11:56.241598 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:56.241556    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p" event={"ID":"a54fd9e3-e867-475a-ad77-09acb484c61e","Type":"ContainerDied","Data":"00d577dd9a77787edc2b669c2b394dc8968cc91892410b172b7870758703213c"}
Jul 28 17:11:56.241636 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:56.241605    2577 scope.go:117] "RemoveContainer" containerID="c84df7475f3c58d13edb8d15810c462c85827733167aabc50b48a75c732adad8"
Jul 28 17:11:56.241881 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:56.241861    2577 scope.go:117] "RemoveContainer" containerID="00d577dd9a77787edc2b669c2b394dc8968cc91892410b172b7870758703213c"
Jul 28 17:11:56.242031 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:56.242010    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-vjx7p_openshift-console-operator(a54fd9e3-e867-475a-ad77-09acb484c61e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p" podUID="a54fd9e3-e867-475a-ad77-09acb484c61e"
Jul 28 17:11:57.245652 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.245622    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:11:57.246025 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.245915    2577 scope.go:117] "RemoveContainer" containerID="00d577dd9a77787edc2b669c2b394dc8968cc91892410b172b7870758703213c"
Jul 28 17:11:57.246082 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:57.246063    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-vjx7p_openshift-console-operator(a54fd9e3-e867-475a-ad77-09acb484c61e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p" podUID="a54fd9e3-e867-475a-ad77-09acb484c61e"
Jul 28 17:11:57.352356 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.352328    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2ql87"]
Jul 28 17:11:57.354753 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.354737    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.362536 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.362515    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:11:57.362651 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.362523    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:11:57.362651 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.362523    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:11:57.362762 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.362649    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:11:57.362762 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.362662    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-nvmsd\""
Jul 28 17:11:57.392804 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.392777    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ql87"]
Jul 28 17:11:57.499698 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.499610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.499698 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.499655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ef96cb6-2f40-4f9a-979a-c32a05499500-data-volume\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.499698 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.499701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6ef96cb6-2f40-4f9a-979a-c32a05499500-crio-socket\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.499904 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.499731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.499904 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.499802    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77hrw\" (UniqueName: \"kubernetes.io/projected/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-api-access-77hrw\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.600806 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.600769    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.600943 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.600836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77hrw\" (UniqueName: \"kubernetes.io/projected/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-api-access-77hrw\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.601006 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:57.600943    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:57.601006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.600996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.601155 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:57.601015    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:58.100993294 +0000 UTC m=+76.744962775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:57.601155 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:57.601043    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:57.601155 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.601067    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ef96cb6-2f40-4f9a-979a-c32a05499500-data-volume\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.601155 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:57.601091    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:58.101075694 +0000 UTC m=+76.745045163 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:57.601155 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.601135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6ef96cb6-2f40-4f9a-979a-c32a05499500-crio-socket\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.601449 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.601231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6ef96cb6-2f40-4f9a-979a-c32a05499500-crio-socket\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.601449 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.601414    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ef96cb6-2f40-4f9a-979a-c32a05499500-data-volume\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.609337 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.609315    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77hrw\" (UniqueName: \"kubernetes.io/projected/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-api-access-77hrw\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:57.811319 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.811230    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx"]
Jul 28 17:11:57.813121 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.813106    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx"
Jul 28 17:11:57.815961 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.815937    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:57.816386 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.816367    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 17:11:57.816552 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.816524    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vtnxt\""
Jul 28 17:11:57.828015 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.827995    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx"]
Jul 28 17:11:57.903367 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:57.903341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjw9\" (UniqueName: \"kubernetes.io/projected/1f91a8ac-7750-4476-a1a9-3a3f49844fab-kube-api-access-bhjw9\") pod \"migrator-6865c9967-9vrjx\" (UID: \"1f91a8ac-7750-4476-a1a9-3a3f49844fab\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx"
Jul 28 17:11:58.003784 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:58.003738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjw9\" (UniqueName: \"kubernetes.io/projected/1f91a8ac-7750-4476-a1a9-3a3f49844fab-kube-api-access-bhjw9\") pod \"migrator-6865c9967-9vrjx\" (UID: \"1f91a8ac-7750-4476-a1a9-3a3f49844fab\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx"
Jul 28 17:11:58.013698 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:58.013670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjw9\" (UniqueName: \"kubernetes.io/projected/1f91a8ac-7750-4476-a1a9-3a3f49844fab-kube-api-access-bhjw9\") pod \"migrator-6865c9967-9vrjx\" (UID: \"1f91a8ac-7750-4476-a1a9-3a3f49844fab\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx"
Jul 28 17:11:58.104824 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:58.104749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:58.104824 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:58.104809    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:58.104978 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:58.104869    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:58.104978 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:58.104932    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:59.104917503 +0000 UTC m=+77.748886970 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:58.104978 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:58.104931    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:58.104978 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:58.104978    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:59.104965166 +0000 UTC m=+77.748934639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:58.123565 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:58.123543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx"
Jul 28 17:11:58.248194 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:58.248163    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx"]
Jul 28 17:11:58.251656 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:11:58.251626    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f91a8ac_7750_4476_a1a9_3a3f49844fab.slice/crio-2609bf44e15de1ec36b53c13753f406773eb2d602b2c98720c825c78ec5c6e40 WatchSource:0}: Error finding container 2609bf44e15de1ec36b53c13753f406773eb2d602b2c98720c825c78ec5c6e40: Status 404 returned error can't find the container with id 2609bf44e15de1ec36b53c13753f406773eb2d602b2c98720c825c78ec5c6e40
Jul 28 17:11:59.112512 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.112476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:59.112708 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.112560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:11:59.112708 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:59.112641    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:59.112708 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:59.112663    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:59.112868 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:59.112722    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:01.112701269 +0000 UTC m=+79.756670741 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:59.112868 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:59.112743    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:01.112733618 +0000 UTC m=+79.756703088 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:59.252942 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.252904    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx" event={"ID":"1f91a8ac-7750-4476-a1a9-3a3f49844fab","Type":"ContainerStarted","Data":"2609bf44e15de1ec36b53c13753f406773eb2d602b2c98720c825c78ec5c6e40"}
Jul 28 17:11:59.469919 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.469895    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mz5qt_c40a55bc-b4ed-4df5-8c3c-a939e7989b12/dns-node-resolver/0.log"
Jul 28 17:11:59.585435 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.585410    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-s57bc"]
Jul 28 17:11:59.587388 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.587372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.590554 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.590530    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 17:11:59.590650 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.590535    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 17:11:59.591701 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.591682    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 17:11:59.591809 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.591719    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mgv5m\""
Jul 28 17:11:59.591809 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.591732    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 17:11:59.600225 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.600201    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-s57bc"]
Jul 28 17:11:59.717128 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.717075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0936c25-b509-4f74-b127-17b17a9b96bb-signing-cabundle\") pod \"service-ca-658d7b7c78-s57bc\" (UID: \"d0936c25-b509-4f74-b127-17b17a9b96bb\") " pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.717218 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.717130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0936c25-b509-4f74-b127-17b17a9b96bb-signing-key\") pod \"service-ca-658d7b7c78-s57bc\" (UID: \"d0936c25-b509-4f74-b127-17b17a9b96bb\") " pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.717218 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.717156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpqw\" (UniqueName: \"kubernetes.io/projected/d0936c25-b509-4f74-b127-17b17a9b96bb-kube-api-access-zgpqw\") pod \"service-ca-658d7b7c78-s57bc\" (UID: \"d0936c25-b509-4f74-b127-17b17a9b96bb\") " pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.817791 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.817761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0936c25-b509-4f74-b127-17b17a9b96bb-signing-cabundle\") pod \"service-ca-658d7b7c78-s57bc\" (UID: \"d0936c25-b509-4f74-b127-17b17a9b96bb\") " pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.817870 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.817819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0936c25-b509-4f74-b127-17b17a9b96bb-signing-key\") pod \"service-ca-658d7b7c78-s57bc\" (UID: \"d0936c25-b509-4f74-b127-17b17a9b96bb\") " pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.817870 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.817844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpqw\" (UniqueName: \"kubernetes.io/projected/d0936c25-b509-4f74-b127-17b17a9b96bb-kube-api-access-zgpqw\") pod \"service-ca-658d7b7c78-s57bc\" (UID: \"d0936c25-b509-4f74-b127-17b17a9b96bb\") " pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.818417 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.818394    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d0936c25-b509-4f74-b127-17b17a9b96bb-signing-cabundle\") pod \"service-ca-658d7b7c78-s57bc\" (UID: \"d0936c25-b509-4f74-b127-17b17a9b96bb\") " pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.820010 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.819991    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d0936c25-b509-4f74-b127-17b17a9b96bb-signing-key\") pod \"service-ca-658d7b7c78-s57bc\" (UID: \"d0936c25-b509-4f74-b127-17b17a9b96bb\") " pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.825836 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.825815    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpqw\" (UniqueName: \"kubernetes.io/projected/d0936c25-b509-4f74-b127-17b17a9b96bb-kube-api-access-zgpqw\") pod \"service-ca-658d7b7c78-s57bc\" (UID: \"d0936c25-b509-4f74-b127-17b17a9b96bb\") " pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.896453 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.896435    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-s57bc"
Jul 28 17:11:59.918954 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.918932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:59.919066 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:11:59.918967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:11:59.919135 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:59.919079    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:12:07.919063634 +0000 UTC m=+86.563033105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:59.919135 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:59.919084    2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:59.919217 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:11:59.919136    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:12:07.919119259 +0000 UTC m=+86.563088731 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : secret "router-metrics-certs-default" not found
Jul 28 17:12:00.015383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:00.015361    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-s57bc"]
Jul 28 17:12:00.017410 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:00.017389    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0936c25_b509_4f74_b127_17b17a9b96bb.slice/crio-d9f9fbe24130f6d6aa897e3e47af98117f5ef930a98eee8079f3444cbf707393 WatchSource:0}: Error finding container d9f9fbe24130f6d6aa897e3e47af98117f5ef930a98eee8079f3444cbf707393: Status 404 returned error can't find the container with id d9f9fbe24130f6d6aa897e3e47af98117f5ef930a98eee8079f3444cbf707393
Jul 28 17:12:00.257168 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:00.257091    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-s57bc" event={"ID":"d0936c25-b509-4f74-b127-17b17a9b96bb","Type":"ContainerStarted","Data":"d9f9fbe24130f6d6aa897e3e47af98117f5ef930a98eee8079f3444cbf707393"}
Jul 28 17:12:00.258609 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:00.258583    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx" event={"ID":"1f91a8ac-7750-4476-a1a9-3a3f49844fab","Type":"ContainerStarted","Data":"63f87003ab1906196262a6669d028272ca080ce67f55597967aa6f16a8b8f8d7"}
Jul 28 17:12:00.258722 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:00.258642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx" event={"ID":"1f91a8ac-7750-4476-a1a9-3a3f49844fab","Type":"ContainerStarted","Data":"80c7e1a54fdeea0708ab30150b5b7f18c214d27995422f8f0568f876c94ecb88"}
Jul 28 17:12:00.277876 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:00.277838    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-9vrjx" podStartSLOduration=2.127742673 podStartE2EDuration="3.277825285s" totalImagesPullingTime="1.150082612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:00.276574862 +0000 UTC m=+78.920544351" watchObservedRunningTime="2026-07-28 17:12:00.277825285 +0000 UTC m=+78.921794771"
Jul 28 17:12:00.475922 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:00.475904    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jht5g_c023b43e-30f9-4d56-aa4f-50f5e86724cf/node-ca/0.log"
Jul 28 17:12:01.126457 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:01.126426    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:12:01.126667 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:01.126508    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:12:01.126667 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:01.126577    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:12:01.126667 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:01.126586    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:12:01.126667 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:01.126635    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:05.126615033 +0000 UTC m=+83.770584501 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : configmap "kube-rbac-proxy" not found
Jul 28 17:12:01.126667 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:01.126649    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:05.126642118 +0000 UTC m=+83.770611585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:12:01.265042 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:01.265000    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wmsx9_01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c/serve-healthcheck-canary/0.log"
Jul 28 17:12:02.265821 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:02.265744    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-s57bc" event={"ID":"d0936c25-b509-4f74-b127-17b17a9b96bb","Type":"ContainerStarted","Data":"3a657a2bbcf9de3849ed7dc34494f54af4cb330b785bc13120fa2c2cc24347a6"}
Jul 28 17:12:02.283234 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:02.283058    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-s57bc" podStartSLOduration=1.355443115 podStartE2EDuration="3.283043886s" totalImagesPullingTime="1.927600771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:02.282837535 +0000 UTC m=+80.926807027" watchObservedRunningTime="2026-07-28 17:12:02.283043886 +0000 UTC m=+80.927013367"
Jul 28 17:12:02.382527 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:02.382488    2577 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:12:02.382668 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:02.382546    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:12:02.382843 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:02.382831    2577 scope.go:117] "RemoveContainer" containerID="00d577dd9a77787edc2b669c2b394dc8968cc91892410b172b7870758703213c"
Jul 28 17:12:02.382971 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:02.382955    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-vjx7p_openshift-console-operator(a54fd9e3-e867-475a-ad77-09acb484c61e)\"" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p" podUID="a54fd9e3-e867-475a-ad77-09acb484c61e"
Jul 28 17:12:05.157540 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:05.157417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:12:05.157540 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:05.157486    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:12:05.157540 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:05.157538    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:12:05.158049 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:05.157593    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:12:05.158049 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:05.157604    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:13.157589181 +0000 UTC m=+91.801558649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : configmap "kube-rbac-proxy" not found
Jul 28 17:12:05.158049 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:05.157631    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls podName:6ef96cb6-2f40-4f9a-979a-c32a05499500 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:13.157620124 +0000 UTC m=+91.801589592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2ql87" (UID: "6ef96cb6-2f40-4f9a-979a-c32a05499500") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:12:07.977281 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:07.977227    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:07.977281 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:07.977286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:07.977834 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:07.977443    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle podName:eb22b701-7530-4ee7-9116-29d6dfde5b0a nodeName:}" failed. No retries permitted until 2026-07-28 17:12:23.977416911 +0000 UTC m=+102.621386596 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle") pod "router-default-64cc6c558f-tgnb7" (UID: "eb22b701-7530-4ee7-9116-29d6dfde5b0a") : configmap references non-existent config key: service-ca.crt
Jul 28 17:12:07.979540 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:07.979511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb22b701-7530-4ee7-9116-29d6dfde5b0a-metrics-certs\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:13.219352 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:13.219308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:12:13.219753 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:13.219377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:12:13.219817 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:13.219801    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ef96cb6-2f40-4f9a-979a-c32a05499500-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:12:13.221518 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:13.221496    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ef96cb6-2f40-4f9a-979a-c32a05499500-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2ql87\" (UID: \"6ef96cb6-2f40-4f9a-979a-c32a05499500\") " pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:12:13.263209 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:13.263186    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2ql87"
Jul 28 17:12:13.383146 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:13.383106    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2ql87"]
Jul 28 17:12:13.385750 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:13.385721    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef96cb6_2f40_4f9a_979a_c32a05499500.slice/crio-7130f378380190a6e5b07563b4ded70048a7de4af6ac383549d82c9f2f4a6fc6 WatchSource:0}: Error finding container 7130f378380190a6e5b07563b4ded70048a7de4af6ac383549d82c9f2f4a6fc6: Status 404 returned error can't find the container with id 7130f378380190a6e5b07563b4ded70048a7de4af6ac383549d82c9f2f4a6fc6
Jul 28 17:12:14.306263 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:14.306164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ql87" event={"ID":"6ef96cb6-2f40-4f9a-979a-c32a05499500","Type":"ContainerStarted","Data":"7130f378380190a6e5b07563b4ded70048a7de4af6ac383549d82c9f2f4a6fc6"}
Jul 28 17:12:15.899735 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:15.899702    2577 scope.go:117] "RemoveContainer" containerID="00d577dd9a77787edc2b669c2b394dc8968cc91892410b172b7870758703213c"
Jul 28 17:12:18.319786 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:18.319752    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ql87" event={"ID":"6ef96cb6-2f40-4f9a-979a-c32a05499500","Type":"ContainerStarted","Data":"6f6324c8d27976d995b8a35ec8a8779025356ec1073da8f4b00ff2c65a00face"}
Jul 28 17:12:18.321593 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:18.321569    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:12:18.321694 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:18.321624    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p" event={"ID":"a54fd9e3-e867-475a-ad77-09acb484c61e","Type":"ContainerStarted","Data":"b79a7b45f386f4368c6071fb1ce55fcdc81c22a0a17c1b12e41a098b8cc6cb81"}
Jul 28 17:12:18.321952 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:18.321922    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:12:18.344844 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:18.344800    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p" podStartSLOduration=25.457652621 podStartE2EDuration="27.344780166s" totalImagesPullingTime="1.887127545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:18.343756484 +0000 UTC m=+96.987725999" watchObservedRunningTime="2026-07-28 17:12:18.344780166 +0000 UTC m=+96.988749656"
Jul 28 17:12:18.623163 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:18.623077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-vjx7p"
Jul 28 17:12:20.680614 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:20.680586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:12:20.683263 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:20.683219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c648388c-85fe-4368-b838-b3fb9f356628-metrics-tls\") pod \"dns-default-d7rfz\" (UID: \"c648388c-85fe-4368-b838-b3fb9f356628\") " pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:12:20.949454 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:20.949383    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-947wh\""
Jul 28 17:12:20.956753 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:20.956735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:12:21.892104 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:21.892074    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d7rfz"]
Jul 28 17:12:21.894492 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:21.894454    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc648388c_85fe_4368_b838_b3fb9f356628.slice/crio-73bb006077da7644ae1712b1f63c2261fac489dec87b965efc45f110d0706cee WatchSource:0}: Error finding container 73bb006077da7644ae1712b1f63c2261fac489dec87b965efc45f110d0706cee: Status 404 returned error can't find the container with id 73bb006077da7644ae1712b1f63c2261fac489dec87b965efc45f110d0706cee
Jul 28 17:12:22.337446 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:22.337405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7rfz" event={"ID":"c648388c-85fe-4368-b838-b3fb9f356628","Type":"ContainerStarted","Data":"73bb006077da7644ae1712b1f63c2261fac489dec87b965efc45f110d0706cee"}
Jul 28 17:12:22.339367 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:22.339335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ql87" event={"ID":"6ef96cb6-2f40-4f9a-979a-c32a05499500","Type":"ContainerStarted","Data":"15c6d021db57094c0161805d66d0002ed05b27530e3e98d2fab4cd3df9343932"}
Jul 28 17:12:24.008220 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:24.008186    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:24.008827 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:24.008806    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eb22b701-7530-4ee7-9116-29d6dfde5b0a-service-ca-bundle\") pod \"router-default-64cc6c558f-tgnb7\" (UID: \"eb22b701-7530-4ee7-9116-29d6dfde5b0a\") " pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:24.233499 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:24.233455    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b5btq"
Jul 28 17:12:24.270023 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:24.269932    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:25.642922 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:25.642894    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64cc6c558f-tgnb7"]
Jul 28 17:12:25.645547 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:25.645524    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb22b701_7530_4ee7_9116_29d6dfde5b0a.slice/crio-31d5b779b8107003625e1c439616bbdea217acaa476b7cba108bcfeeaab7fc88 WatchSource:0}: Error finding container 31d5b779b8107003625e1c439616bbdea217acaa476b7cba108bcfeeaab7fc88: Status 404 returned error can't find the container with id 31d5b779b8107003625e1c439616bbdea217acaa476b7cba108bcfeeaab7fc88
Jul 28 17:12:26.358517 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:26.358482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2ql87" event={"ID":"6ef96cb6-2f40-4f9a-979a-c32a05499500","Type":"ContainerStarted","Data":"e9c702e839cdb66b0a61dcd4b7b26408cee5706f853676306caff38756ddef77"}
Jul 28 17:12:26.359902 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:26.359873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64cc6c558f-tgnb7" event={"ID":"eb22b701-7530-4ee7-9116-29d6dfde5b0a","Type":"ContainerStarted","Data":"58f973c4fc17a5e6ee3ddf4cc5aaaff3a35c6563e38419414f4bc352b752d94d"}
Jul 28 17:12:26.360009 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:26.359917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64cc6c558f-tgnb7" event={"ID":"eb22b701-7530-4ee7-9116-29d6dfde5b0a","Type":"ContainerStarted","Data":"31d5b779b8107003625e1c439616bbdea217acaa476b7cba108bcfeeaab7fc88"}
Jul 28 17:12:26.361432 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:26.361409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7rfz" event={"ID":"c648388c-85fe-4368-b838-b3fb9f356628","Type":"ContainerStarted","Data":"e5136acf932d22d95c22494d0cd97ffb0ea643b3a5a2f334ad13478c893c1c0f"}
Jul 28 17:12:26.361522 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:26.361446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d7rfz" event={"ID":"c648388c-85fe-4368-b838-b3fb9f356628","Type":"ContainerStarted","Data":"4873d8c3f4176004ec81f5a310535f1410a40aff3aa6aeb59ce20b2d41c01dc5"}
Jul 28 17:12:26.361576 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:26.361556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:12:26.379167 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:26.379130    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2ql87" podStartSLOduration=17.054138609 podStartE2EDuration="29.379119082s" totalImagesPullingTime="12.324980473s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:26.377468714 +0000 UTC m=+105.021438203" watchObservedRunningTime="2026-07-28 17:12:26.379119082 +0000 UTC m=+105.023088571"
Jul 28 17:12:26.399105 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:26.399063    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64cc6c558f-tgnb7" podStartSLOduration=34.399048844 podStartE2EDuration="34.399048844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:26.398380111 +0000 UTC m=+105.042349600" watchObservedRunningTime="2026-07-28 17:12:26.399048844 +0000 UTC m=+105.043018335"
Jul 28 17:12:26.416832 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:26.416798    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d7rfz" podStartSLOduration=66.792454494 podStartE2EDuration="1m10.416787553s" totalImagesPullingTime="3.624333059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:26.416317351 +0000 UTC m=+105.060286840" watchObservedRunningTime="2026-07-28 17:12:26.416787553 +0000 UTC m=+105.060757043"
Jul 28 17:12:27.270568 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:27.270536    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:27.273164 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:27.273146    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:27.365013 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:27.364989    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:27.365858 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:27.365841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64cc6c558f-tgnb7"
Jul 28 17:12:28.811336 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:28.811303    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"]
Jul 28 17:12:28.840370 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:28.840337    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"]
Jul 28 17:12:28.840533 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:28.840442    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"
Jul 28 17:12:28.843298 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:28.843282    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 17:12:28.843422 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:28.843387    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-k6zf2\""
Jul 28 17:12:28.942619 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:28.942586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3932887-752b-4af2-988d-dff0abd73767-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-t9dcf\" (UID: \"b3932887-752b-4af2-988d-dff0abd73767\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"
Jul 28 17:12:29.043133 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:29.043098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3932887-752b-4af2-988d-dff0abd73767-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-t9dcf\" (UID: \"b3932887-752b-4af2-988d-dff0abd73767\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"
Jul 28 17:12:29.043294 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:29.043262    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 17:12:29.043336 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:29.043327    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3932887-752b-4af2-988d-dff0abd73767-tls-certificates podName:b3932887-752b-4af2-988d-dff0abd73767 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:29.543310837 +0000 UTC m=+108.187280305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b3932887-752b-4af2-988d-dff0abd73767-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-t9dcf" (UID: "b3932887-752b-4af2-988d-dff0abd73767") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 17:12:29.547130 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:29.547095    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3932887-752b-4af2-988d-dff0abd73767-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-t9dcf\" (UID: \"b3932887-752b-4af2-988d-dff0abd73767\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"
Jul 28 17:12:29.549530 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:29.549504    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b3932887-752b-4af2-988d-dff0abd73767-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-t9dcf\" (UID: \"b3932887-752b-4af2-988d-dff0abd73767\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"
Jul 28 17:12:29.750042 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:29.750005    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"
Jul 28 17:12:29.883982 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:29.883950    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"]
Jul 28 17:12:29.886911 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:29.886881    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3932887_752b_4af2_988d_dff0abd73767.slice/crio-7b9df31854a4a45b01c2b89db8e21aabef4534017e1ab4a34382f11fbd2d1e31 WatchSource:0}: Error finding container 7b9df31854a4a45b01c2b89db8e21aabef4534017e1ab4a34382f11fbd2d1e31: Status 404 returned error can't find the container with id 7b9df31854a4a45b01c2b89db8e21aabef4534017e1ab4a34382f11fbd2d1e31
Jul 28 17:12:30.380317 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:30.380277    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf" event={"ID":"b3932887-752b-4af2-988d-dff0abd73767","Type":"ContainerStarted","Data":"7b9df31854a4a45b01c2b89db8e21aabef4534017e1ab4a34382f11fbd2d1e31"}
Jul 28 17:12:31.384500 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.384470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf" event={"ID":"b3932887-752b-4af2-988d-dff0abd73767","Type":"ContainerStarted","Data":"2534159374c6c15fd3ffaf6b9766fdcecdcad03e276676038a80e8761c1842c8"}
Jul 28 17:12:31.384932 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.384696    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"
Jul 28 17:12:31.389184 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.389159    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf"
Jul 28 17:12:31.403442 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.403400    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-t9dcf" podStartSLOduration=2.384602418 podStartE2EDuration="3.403385919s" totalImagesPullingTime="1.018783501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:31.402701184 +0000 UTC m=+110.046670675" watchObservedRunningTime="2026-07-28 17:12:31.403385919 +0000 UTC m=+110.047355409"
Jul 28 17:12:31.884691 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.884666    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-545bt"]
Jul 28 17:12:31.886691 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.886675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:31.889625 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.889598    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:12:31.889746 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.889672    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 17:12:31.889746 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.889709    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 17:12:31.891176 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.891005    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xkbx4\""
Jul 28 17:12:31.891176 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.891063    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:12:31.891176 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.891094    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:12:31.904076 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:31.904052    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-545bt"]
Jul 28 17:12:32.063981 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.063956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxbk\" (UniqueName: \"kubernetes.io/projected/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-kube-api-access-dlxbk\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.064122 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.063999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.064122 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.064028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.064122 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.064085    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.165401 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.165336    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxbk\" (UniqueName: \"kubernetes.io/projected/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-kube-api-access-dlxbk\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.165514 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.165398    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.165514 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.165446    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.165514 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.165473    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.165669 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:32.165555    2577 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 17:12:32.165669 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:32.165625    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-prometheus-operator-tls podName:13a8e82c-5c0c-4dd7-b155-08ac4c3927eb nodeName:}" failed. No retries permitted until 2026-07-28 17:12:32.665605608 +0000 UTC m=+111.309575081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-545bt" (UID: "13a8e82c-5c0c-4dd7-b155-08ac4c3927eb") : secret "prometheus-operator-tls" not found
Jul 28 17:12:32.166226 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.166206    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.167874 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.167852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.176460 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.176440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxbk\" (UniqueName: \"kubernetes.io/projected/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-kube-api-access-dlxbk\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.669030 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.669006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.671309 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.671290    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/13a8e82c-5c0c-4dd7-b155-08ac4c3927eb-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-545bt\" (UID: \"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.795590 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.795561    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt"
Jul 28 17:12:32.920282 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:32.917314    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-545bt"]
Jul 28 17:12:32.923950 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:32.923926    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13a8e82c_5c0c_4dd7_b155_08ac4c3927eb.slice/crio-5983bdfd2219e232ab647e7c24c30023c450b7a03855ce4e0bb09acbd0105ba6 WatchSource:0}: Error finding container 5983bdfd2219e232ab647e7c24c30023c450b7a03855ce4e0bb09acbd0105ba6: Status 404 returned error can't find the container with id 5983bdfd2219e232ab647e7c24c30023c450b7a03855ce4e0bb09acbd0105ba6
Jul 28 17:12:33.393822 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:33.393779    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt" event={"ID":"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb","Type":"ContainerStarted","Data":"5983bdfd2219e232ab647e7c24c30023c450b7a03855ce4e0bb09acbd0105ba6"}
Jul 28 17:12:34.399432 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:34.399407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt" event={"ID":"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb","Type":"ContainerStarted","Data":"c8703dbdee5f43c5a0ae88393e7d2769cd351aa119a84a180ea41f53db264dd3"}
Jul 28 17:12:34.399769 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:34.399456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt" event={"ID":"13a8e82c-5c0c-4dd7-b155-08ac4c3927eb","Type":"ContainerStarted","Data":"d8835fe2e6f09baad9a50c36a6a14b5310d9638d196c9ceaa6be72ac7792c6a9"}
Jul 28 17:12:34.419915 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:34.419873    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-545bt" podStartSLOduration=2.3269632639999998 podStartE2EDuration="3.419859233s" totalImagesPullingTime="1.092895969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:34.418485492 +0000 UTC m=+113.062454982" watchObservedRunningTime="2026-07-28 17:12:34.419859233 +0000 UTC m=+113.063828723"
Jul 28 17:12:36.270815 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.270782    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"]
Jul 28 17:12:36.276494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.276470    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.279396 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.279375    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 17:12:36.279508 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.279459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:12:36.279627 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.279611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7fk65\""
Jul 28 17:12:36.287180 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.287157    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"]
Jul 28 17:12:36.293744 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.293717    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.293868 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.293759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.293868 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.293839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.293990 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.293918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxms\" (UniqueName: \"kubernetes.io/projected/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-kube-api-access-dxxms\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.296014 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.295991    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fwgpv"]
Jul 28 17:12:36.300204 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.300188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.302959 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.302787    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:12:36.303309 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.303287    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-44md9\""
Jul 28 17:12:36.303428 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.303309    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:12:36.303579 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.303378    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:12:36.366750 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.366730    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d7rfz"
Jul 28 17:12:36.395210 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm4m9\" (UniqueName: \"kubernetes.io/projected/f7120024-8fa9-4cd3-8e78-81df46bd8680-kube-api-access-vm4m9\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.395329 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395240    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.395329 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.395329 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395310    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-textfile\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.395465 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7120024-8fa9-4cd3-8e78-81df46bd8680-sys\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.395465 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:36.395391    2577 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 17:12:36.395465 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395401    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7120024-8fa9-4cd3-8e78-81df46bd8680-root\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.395465 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-tls\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.395465 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:36.395459    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-openshift-state-metrics-tls podName:4d125b48-7cb2-4a74-9a60-e6549aaea6d4 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:36.895434863 +0000 UTC m=+115.539404336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-rt26j" (UID: "4d125b48-7cb2-4a74-9a60-e6549aaea6d4") : secret "openshift-state-metrics-tls" not found
Jul 28 17:12:36.395663 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-wtmp\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.395663 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxms\" (UniqueName: \"kubernetes.io/projected/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-kube-api-access-dxxms\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.395663 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7120024-8fa9-4cd3-8e78-81df46bd8680-metrics-client-ca\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.395968 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.395650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.396358 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.396021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.396436 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.396382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-accelerators-collector-config\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.398023 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.397996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.412315 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.411036    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.417272 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.416839    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxms\" (UniqueName: \"kubernetes.io/projected/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-kube-api-access-dxxms\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.497555 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7120024-8fa9-4cd3-8e78-81df46bd8680-sys\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.497738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497567    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7120024-8fa9-4cd3-8e78-81df46bd8680-root\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.497738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-tls\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.497738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497635    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-wtmp\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.497738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f7120024-8fa9-4cd3-8e78-81df46bd8680-sys\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.497738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497663    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7120024-8fa9-4cd3-8e78-81df46bd8680-metrics-client-ca\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.497738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f7120024-8fa9-4cd3-8e78-81df46bd8680-root\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.497738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497696    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-accelerators-collector-config\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.497738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm4m9\" (UniqueName: \"kubernetes.io/projected/f7120024-8fa9-4cd3-8e78-81df46bd8680-kube-api-access-vm4m9\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.498133 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:36.497750    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:12:36.498133 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497785    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.498133 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:36.497813    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-tls podName:f7120024-8fa9-4cd3-8e78-81df46bd8680 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:36.997793742 +0000 UTC m=+115.641763222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-tls") pod "node-exporter-fwgpv" (UID: "f7120024-8fa9-4cd3-8e78-81df46bd8680") : secret "node-exporter-tls" not found
Jul 28 17:12:36.498133 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-wtmp\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.498133 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.497872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-textfile\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.498409 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.498212    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-textfile\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.498533 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.498510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-accelerators-collector-config\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.498590 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.498533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7120024-8fa9-4cd3-8e78-81df46bd8680-metrics-client-ca\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.500424 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.500405    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.509651 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.509629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm4m9\" (UniqueName: \"kubernetes.io/projected/f7120024-8fa9-4cd3-8e78-81df46bd8680-kube-api-access-vm4m9\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:36.515342 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.515322    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b76b7f848-9rz5g"]
Jul 28 17:12:36.520710 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.520693    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.523993 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.523946    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:12:36.524180 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.524164    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:12:36.524348 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.524269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:12:36.524416 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.524393    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:12:36.524596 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.524305    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:12:36.524671 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.524637    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:12:36.525219 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.525202    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9b9mq\""
Jul 28 17:12:36.525324 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.525308    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:12:36.531211 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.531191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:12:36.533515 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.533497    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b76b7f848-9rz5g"]
Jul 28 17:12:36.598484 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.598459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-service-ca\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.598602 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.598530    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-serving-cert\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.598642 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.598601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-oauth-config\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.598684 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.598657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-trusted-ca-bundle\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.598720 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.598708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-config\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.598750 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.598732    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8mqq\" (UniqueName: \"kubernetes.io/projected/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-kube-api-access-c8mqq\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.598784 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.598776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-oauth-serving-cert\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.699723 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.699695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-serving-cert\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.699863 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.699746    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-oauth-config\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.699932 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.699903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-trusted-ca-bundle\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.699993 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.699953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-config\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.699993 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.699979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8mqq\" (UniqueName: \"kubernetes.io/projected/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-kube-api-access-c8mqq\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.700087 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.700019    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-oauth-serving-cert\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.700087 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.700046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-service-ca\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.700687 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.700664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-service-ca\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.700790 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.700665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-config\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.700940 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.700919    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-oauth-serving-cert\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.701113 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.701090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-trusted-ca-bundle\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.702649 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.702627    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-oauth-config\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.702739 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.702659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-serving-cert\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.710626 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.710606    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8mqq\" (UniqueName: \"kubernetes.io/projected/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-kube-api-access-c8mqq\") pod \"console-5b76b7f848-9rz5g\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") " pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.831845 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.831786    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:36.901502 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.901469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.903562 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.903539    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d125b48-7cb2-4a74-9a60-e6549aaea6d4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-rt26j\" (UID: \"4d125b48-7cb2-4a74-9a60-e6549aaea6d4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:36.958982 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:36.958938    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b76b7f848-9rz5g"]
Jul 28 17:12:36.961417 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:36.961390    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a5c68d_f9b9_42c3_b08f_17e90a456cc1.slice/crio-4ce63e93f54ccab35191d2af6c1f9b8631c17a3487c8778467e7719e030bb0a3 WatchSource:0}: Error finding container 4ce63e93f54ccab35191d2af6c1f9b8631c17a3487c8778467e7719e030bb0a3: Status 404 returned error can't find the container with id 4ce63e93f54ccab35191d2af6c1f9b8631c17a3487c8778467e7719e030bb0a3
Jul 28 17:12:37.002559 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:37.002536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-tls\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:37.004554 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:37.004537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f7120024-8fa9-4cd3-8e78-81df46bd8680-node-exporter-tls\") pod \"node-exporter-fwgpv\" (UID: \"f7120024-8fa9-4cd3-8e78-81df46bd8680\") " pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:37.186336 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:37.186310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"
Jul 28 17:12:37.211125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:37.211101    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fwgpv"
Jul 28 17:12:37.223596 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:37.223568    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7120024_8fa9_4cd3_8e78_81df46bd8680.slice/crio-af8387ce2e28c098bb324ce85aa50e8b4009d22f6291f0fd7aab1b68681f132f WatchSource:0}: Error finding container af8387ce2e28c098bb324ce85aa50e8b4009d22f6291f0fd7aab1b68681f132f: Status 404 returned error can't find the container with id af8387ce2e28c098bb324ce85aa50e8b4009d22f6291f0fd7aab1b68681f132f
Jul 28 17:12:37.324909 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:37.324843    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j"]
Jul 28 17:12:37.328358 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:37.328300    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d125b48_7cb2_4a74_9a60_e6549aaea6d4.slice/crio-131c4e629918df3d85df9b17fb9862c53473b3ab1cd3c16e21529036a1a63f6e WatchSource:0}: Error finding container 131c4e629918df3d85df9b17fb9862c53473b3ab1cd3c16e21529036a1a63f6e: Status 404 returned error can't find the container with id 131c4e629918df3d85df9b17fb9862c53473b3ab1cd3c16e21529036a1a63f6e
Jul 28 17:12:37.421959 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:37.421927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j" event={"ID":"4d125b48-7cb2-4a74-9a60-e6549aaea6d4","Type":"ContainerStarted","Data":"2e36f3b006d32f1f3034586e3dc50197f1ec80f9c5bbb777c17a2016c9f86411"}
Jul 28 17:12:37.422083 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:37.422009    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j" event={"ID":"4d125b48-7cb2-4a74-9a60-e6549aaea6d4","Type":"ContainerStarted","Data":"131c4e629918df3d85df9b17fb9862c53473b3ab1cd3c16e21529036a1a63f6e"}
Jul 28 17:12:37.424590 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:37.424559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fwgpv" event={"ID":"f7120024-8fa9-4cd3-8e78-81df46bd8680","Type":"ContainerStarted","Data":"af8387ce2e28c098bb324ce85aa50e8b4009d22f6291f0fd7aab1b68681f132f"}
Jul 28 17:12:37.426422 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:37.426335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b76b7f848-9rz5g" event={"ID":"96a5c68d-f9b9-42c3-b08f-17e90a456cc1","Type":"ContainerStarted","Data":"4ce63e93f54ccab35191d2af6c1f9b8631c17a3487c8778467e7719e030bb0a3"}
Jul 28 17:12:38.231774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.231741    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"]
Jul 28 17:12:38.237700 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.237675    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.240763 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.240738    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 17:12:38.241013 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.240995    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 17:12:38.241778 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.241742    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-kd6mf\""
Jul 28 17:12:38.245709 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.241992    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3nia91t9g9gcm\""
Jul 28 17:12:38.245709 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.242210    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 17:12:38.245709 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.242430    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 17:12:38.245709 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.245505    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 17:12:38.248608 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.248526    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"]
Jul 28 17:12:38.315768 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.315728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.315768 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.315768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.315986 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.315803    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/644a33d5-946e-4d11-a208-323e444fe48b-metrics-client-ca\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.315986 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.315850    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z62s\" (UniqueName: \"kubernetes.io/projected/644a33d5-946e-4d11-a208-323e444fe48b-kube-api-access-4z62s\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.315986 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.315886    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-tls\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.315986 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.315918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-grpc-tls\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.315986 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.315955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.316232 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.315986    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.416874 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.416832    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.417770 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.416908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.417770 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.417144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.417770 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.417176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.417770 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.417223    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/644a33d5-946e-4d11-a208-323e444fe48b-metrics-client-ca\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.418193 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.418097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/644a33d5-946e-4d11-a208-323e444fe48b-metrics-client-ca\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.418323 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.418293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z62s\" (UniqueName: \"kubernetes.io/projected/644a33d5-946e-4d11-a208-323e444fe48b-kube-api-access-4z62s\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.418409 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.418385    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-tls\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.418598 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.418580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-grpc-tls\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.424836 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.422175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.424836 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.422396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-grpc-tls\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.428136 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.428097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.429311 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.428893    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.429505 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.429477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-tls\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.429799 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.429760    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z62s\" (UniqueName: \"kubernetes.io/projected/644a33d5-946e-4d11-a208-323e444fe48b-kube-api-access-4z62s\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.430149 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.430097    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/644a33d5-946e-4d11-a208-323e444fe48b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5d4cbb8df6-2ljld\" (UID: \"644a33d5-946e-4d11-a208-323e444fe48b\") " pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.433910 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.433881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j" event={"ID":"4d125b48-7cb2-4a74-9a60-e6549aaea6d4","Type":"ContainerStarted","Data":"fa113eb8584a8171a9f7ed3f574cdbf1b3d2207c878c3ec47214a16de5c9cff8"}
Jul 28 17:12:38.435761 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.435735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fwgpv" event={"ID":"f7120024-8fa9-4cd3-8e78-81df46bd8680","Type":"ContainerStarted","Data":"e7a46687f16e9244356e151766aced3e95ffd7f19ae19d9e7aef08feedd8d871"}
Jul 28 17:12:38.551701 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.551656    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:38.768895 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:38.768798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"]
Jul 28 17:12:38.770725 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:38.770674    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644a33d5_946e_4d11_a208_323e444fe48b.slice/crio-fd97b1bccd3e4d1a9b6d4361faff43e87a827eef162f4b8c7925088c029411d5 WatchSource:0}: Error finding container fd97b1bccd3e4d1a9b6d4361faff43e87a827eef162f4b8c7925088c029411d5: Status 404 returned error can't find the container with id fd97b1bccd3e4d1a9b6d4361faff43e87a827eef162f4b8c7925088c029411d5
Jul 28 17:12:39.442561 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:39.442226    2577 generic.go:356] "Generic (PLEG): container finished" podID="f7120024-8fa9-4cd3-8e78-81df46bd8680" containerID="e7a46687f16e9244356e151766aced3e95ffd7f19ae19d9e7aef08feedd8d871" exitCode=0
Jul 28 17:12:39.442561 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:39.442294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fwgpv" event={"ID":"f7120024-8fa9-4cd3-8e78-81df46bd8680","Type":"ContainerDied","Data":"e7a46687f16e9244356e151766aced3e95ffd7f19ae19d9e7aef08feedd8d871"}
Jul 28 17:12:39.443715 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:39.443692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld" event={"ID":"644a33d5-946e-4d11-a208-323e444fe48b","Type":"ContainerStarted","Data":"fd97b1bccd3e4d1a9b6d4361faff43e87a827eef162f4b8c7925088c029411d5"}
Jul 28 17:12:39.446096 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:39.446052    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j" event={"ID":"4d125b48-7cb2-4a74-9a60-e6549aaea6d4","Type":"ContainerStarted","Data":"e80ccd4b511b831bfa30c20c83f3bd26c505404d2f718f774ed78e857985ba8d"}
Jul 28 17:12:39.504865 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:39.504812    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-rt26j" podStartSLOduration=2.321245559 podStartE2EDuration="3.504790574s" totalImagesPullingTime="1.183545015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:39.503664165 +0000 UTC m=+118.147633656" watchObservedRunningTime="2026-07-28 17:12:39.504790574 +0000 UTC m=+118.148760065"
Jul 28 17:12:40.454813 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:40.454776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fwgpv" event={"ID":"f7120024-8fa9-4cd3-8e78-81df46bd8680","Type":"ContainerStarted","Data":"8315a2f3899ce73e1ff90c15292cc1ee51ed991ebbd146228c5aa14eca92a277"}
Jul 28 17:12:40.455234 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:40.454839    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fwgpv" event={"ID":"f7120024-8fa9-4cd3-8e78-81df46bd8680","Type":"ContainerStarted","Data":"61e7c87b879c7c0f75e602e9ddf80f6466c663a318ab847c8d77814cd2df740f"}
Jul 28 17:12:40.456727 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:40.456701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b76b7f848-9rz5g" event={"ID":"96a5c68d-f9b9-42c3-b08f-17e90a456cc1","Type":"ContainerStarted","Data":"dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483"}
Jul 28 17:12:40.478333 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:40.478280    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fwgpv" podStartSLOduration=3.296193866 podStartE2EDuration="4.478238204s" totalImagesPullingTime="1.095256081s" totalInitContainerRuntime="86.788257ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:40.475620412 +0000 UTC m=+119.119589903" watchObservedRunningTime="2026-07-28 17:12:40.478238204 +0000 UTC m=+119.122207695"
Jul 28 17:12:40.494862 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:40.494822    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b76b7f848-9rz5g" podStartSLOduration=1.453642318 podStartE2EDuration="4.4948087s" totalImagesPullingTime="3.041166382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:40.494306555 +0000 UTC m=+119.138276055" watchObservedRunningTime="2026-07-28 17:12:40.4948087 +0000 UTC m=+119.138778190"
Jul 28 17:12:41.012271 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.012156    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"]
Jul 28 17:12:41.016273 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.016163    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:41.019477 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.019449    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cmn2x\""
Jul 28 17:12:41.019707 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.019682    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 17:12:41.025754 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.025724    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"]
Jul 28 17:12:41.043900 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.043866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/337e7f5a-653e-4f40-a755-f57bae2e48b1-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-k4blr\" (UID: \"337e7f5a-653e-4f40-a755-f57bae2e48b1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:41.044020 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.043914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlcv6\" (UniqueName: \"kubernetes.io/projected/337e7f5a-653e-4f40-a755-f57bae2e48b1-kube-api-access-mlcv6\") pod \"monitoring-plugin-68b7fc9574-k4blr\" (UID: \"337e7f5a-653e-4f40-a755-f57bae2e48b1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:41.144875 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.144838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/337e7f5a-653e-4f40-a755-f57bae2e48b1-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-k4blr\" (UID: \"337e7f5a-653e-4f40-a755-f57bae2e48b1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:41.145036 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.144902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlcv6\" (UniqueName: \"kubernetes.io/projected/337e7f5a-653e-4f40-a755-f57bae2e48b1-kube-api-access-mlcv6\") pod \"monitoring-plugin-68b7fc9574-k4blr\" (UID: \"337e7f5a-653e-4f40-a755-f57bae2e48b1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:41.145414 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:41.145393    2577 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 17:12:41.145487 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:12:41.145464    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/337e7f5a-653e-4f40-a755-f57bae2e48b1-monitoring-plugin-cert podName:337e7f5a-653e-4f40-a755-f57bae2e48b1 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:41.645442287 +0000 UTC m=+120.289411758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/337e7f5a-653e-4f40-a755-f57bae2e48b1-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-k4blr" (UID: "337e7f5a-653e-4f40-a755-f57bae2e48b1") : secret "monitoring-plugin-cert" not found
Jul 28 17:12:41.153946 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.153920    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlcv6\" (UniqueName: \"kubernetes.io/projected/337e7f5a-653e-4f40-a755-f57bae2e48b1-kube-api-access-mlcv6\") pod \"monitoring-plugin-68b7fc9574-k4blr\" (UID: \"337e7f5a-653e-4f40-a755-f57bae2e48b1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:41.649385 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.649358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/337e7f5a-653e-4f40-a755-f57bae2e48b1-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-k4blr\" (UID: \"337e7f5a-653e-4f40-a755-f57bae2e48b1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:41.651611 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.651585    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/337e7f5a-653e-4f40-a755-f57bae2e48b1-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-k4blr\" (UID: \"337e7f5a-653e-4f40-a755-f57bae2e48b1\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:41.935110 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.935085    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cmn2x\""
Jul 28 17:12:41.943367 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:41.943346    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:42.073992 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.073969    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"]
Jul 28 17:12:42.076293 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:42.076267    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod337e7f5a_653e_4f40_a755_f57bae2e48b1.slice/crio-3777ce645efc61a8eb37b9057002c6f45ea4ce6b5e457d0df319b5d7d2f44c4d WatchSource:0}: Error finding container 3777ce645efc61a8eb37b9057002c6f45ea4ce6b5e457d0df319b5d7d2f44c4d: Status 404 returned error can't find the container with id 3777ce645efc61a8eb37b9057002c6f45ea4ce6b5e457d0df319b5d7d2f44c4d
Jul 28 17:12:42.468381 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.468336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld" event={"ID":"644a33d5-946e-4d11-a208-323e444fe48b","Type":"ContainerStarted","Data":"30a8a85fd7667909f72e7a5c868dea3ea28f23a608a23193390c65e9faa19969"}
Jul 28 17:12:42.468524 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.468434    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld" event={"ID":"644a33d5-946e-4d11-a208-323e444fe48b","Type":"ContainerStarted","Data":"02b07d30c13e823e4b961c483dcf7e0e8c0aced7df0eaa21354d060fdf271dc8"}
Jul 28 17:12:42.468524 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.468478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld" event={"ID":"644a33d5-946e-4d11-a208-323e444fe48b","Type":"ContainerStarted","Data":"ea8828b4a3aa9dfe1d823236b7b5392ecfa4e5f15eb22ac00470772abe92ec44"}
Jul 28 17:12:42.469484 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.469464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr" event={"ID":"337e7f5a-653e-4f40-a755-f57bae2e48b1","Type":"ContainerStarted","Data":"3777ce645efc61a8eb37b9057002c6f45ea4ce6b5e457d0df319b5d7d2f44c4d"}
Jul 28 17:12:42.488147 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.487299    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:12:42.491063 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.491045    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.494049 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.494029    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 17:12:42.494569 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.494357    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 17:12:42.494569 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.494425    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 17:12:42.494569 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.494446    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 17:12:42.494569 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.494452    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:12:42.494899 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.494881    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7hhpi3bk7vtek\""
Jul 28 17:12:42.495625 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.495419    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 17:12:42.495625 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.495457    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 17:12:42.495625 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.495473    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 17:12:42.495625 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.495459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5k7vj\""
Jul 28 17:12:42.496129 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.496012    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 17:12:42.496221 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.496129    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 17:12:42.496221 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.496142    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 17:12:42.499517 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.499494    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 17:12:42.502859 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.499737    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 17:12:42.508235 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.508208    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 17:12:42.509433 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.509410    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 17:12:42.509534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.509442    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.557900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp5fr\" (UniqueName: \"kubernetes.io/projected/b221fe5e-9ba3-4b22-b088-2274244ab31f-kube-api-access-bp5fr\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.557983    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558016    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b221fe5e-9ba3-4b22-b088-2274244ab31f-config-out\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558079    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b221fe5e-9ba3-4b22-b088-2274244ab31f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558122    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558187    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558235    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558343    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558365    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-config\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.559182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558421    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.560007 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558447    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.560007 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558468    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.560007 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.560007 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.558545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-web-config\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.659818 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.659792    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.659836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b221fe5e-9ba3-4b22-b088-2274244ab31f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.659860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.659885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.659921    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.659950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.659996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660045    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-config\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660150    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-web-config\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660333    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp5fr\" (UniqueName: \"kubernetes.io/projected/b221fe5e-9ba3-4b22-b088-2274244ab31f-kube-api-access-bp5fr\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.660805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.660389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b221fe5e-9ba3-4b22-b088-2274244ab31f-config-out\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.663643 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.661465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.663643 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.661981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.663643 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.662396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.664583 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.664556    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.667653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.665035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.667653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.666419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b221fe5e-9ba3-4b22-b088-2274244ab31f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.667653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.667211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.668827 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.668179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.668827 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.668549    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b221fe5e-9ba3-4b22-b088-2274244ab31f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.669463 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.669134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b221fe5e-9ba3-4b22-b088-2274244ab31f-config-out\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.670288 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.670218    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.670288 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.670277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-web-config\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.670431 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.670238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.670912 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.670764    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.672495 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.672424    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.674631 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.674491    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.674631 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.674583    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.675039 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.674858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-config\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.676774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.676642    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b221fe5e-9ba3-4b22-b088-2274244ab31f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.678468 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.677319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp5fr\" (UniqueName: \"kubernetes.io/projected/b221fe5e-9ba3-4b22-b088-2274244ab31f-kube-api-access-bp5fr\") pod \"prometheus-k8s-0\" (UID: \"b221fe5e-9ba3-4b22-b088-2274244ab31f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.806437 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.806394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:42.976734 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:42.976666    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:12:42.977965 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:42.977941    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb221fe5e_9ba3_4b22_b088_2274244ab31f.slice/crio-930075d76f0f12f0b4eb3db67202b3d440490a47f13db01c7d51e071ea134893 WatchSource:0}: Error finding container 930075d76f0f12f0b4eb3db67202b3d440490a47f13db01c7d51e071ea134893: Status 404 returned error can't find the container with id 930075d76f0f12f0b4eb3db67202b3d440490a47f13db01c7d51e071ea134893
Jul 28 17:12:43.476992 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:43.476954    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld" event={"ID":"644a33d5-946e-4d11-a208-323e444fe48b","Type":"ContainerStarted","Data":"3040723a4643d76187d0d0637365a57da967a2ff8a171e0b4561eb4a548e383a"}
Jul 28 17:12:43.477255 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:43.477088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld" event={"ID":"644a33d5-946e-4d11-a208-323e444fe48b","Type":"ContainerStarted","Data":"b45bdf9f37e825e9dbfefb9f11377664ee10980341735d9cc0cbe431b1c30e9c"}
Jul 28 17:12:43.477255 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:43.477168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld" event={"ID":"644a33d5-946e-4d11-a208-323e444fe48b","Type":"ContainerStarted","Data":"9ed0b83557d33364e9b51b3cf3e6ef6b267d3cad1b067cf3c9f39adb835861de"}
Jul 28 17:12:43.477411 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:43.477268    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:43.480180 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:43.480158    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b221fe5e-9ba3-4b22-b088-2274244ab31f","Type":"ContainerStarted","Data":"930075d76f0f12f0b4eb3db67202b3d440490a47f13db01c7d51e071ea134893"}
Jul 28 17:12:43.504884 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:43.504596    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld" podStartSLOduration=1.941160421 podStartE2EDuration="5.504577486s" totalImagesPullingTime="3.563417065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:43.502073008 +0000 UTC m=+122.146042500" watchObservedRunningTime="2026-07-28 17:12:43.504577486 +0000 UTC m=+122.148546976"
Jul 28 17:12:44.485521 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:44.485481    2577 generic.go:356] "Generic (PLEG): container finished" podID="b221fe5e-9ba3-4b22-b088-2274244ab31f" containerID="29d3ce9f68ffa732da718570d3d053025ed11b766108f14193c44c56e185be78" exitCode=0
Jul 28 17:12:44.485916 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:44.485562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b221fe5e-9ba3-4b22-b088-2274244ab31f","Type":"ContainerDied","Data":"29d3ce9f68ffa732da718570d3d053025ed11b766108f14193c44c56e185be78"}
Jul 28 17:12:44.489477 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:44.489450    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr" event={"ID":"337e7f5a-653e-4f40-a755-f57bae2e48b1","Type":"ContainerStarted","Data":"f0f979873e3c18bc88914b698c77a74702fe46bbe831af9b0aafff3abdda3dd0"}
Jul 28 17:12:44.489861 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:44.489831    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:44.495173 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:44.495149    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr"
Jul 28 17:12:44.534851 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:44.534807    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-k4blr" podStartSLOduration=3.193863554 podStartE2EDuration="4.53479461s" totalImagesPullingTime="1.340931056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:44.534235054 +0000 UTC m=+123.178204544" watchObservedRunningTime="2026-07-28 17:12:44.53479461 +0000 UTC m=+123.178764152"
Jul 28 17:12:46.832743 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:46.832700    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:46.833188 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:46.832779    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:46.838222 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:46.838198    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:47.510571 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:47.510344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b221fe5e-9ba3-4b22-b088-2274244ab31f","Type":"ContainerStarted","Data":"7fff1bd5d15ef2acd8e8fa9775b09b16f6c43ff20f7a7619bad247a23106deae"}
Jul 28 17:12:47.510571 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:47.510459    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b221fe5e-9ba3-4b22-b088-2274244ab31f","Type":"ContainerStarted","Data":"5939532fc06e349e709e98b7f9fa84b7c6d1b9e871a898520190d3424a8bf9eb"}
Jul 28 17:12:47.510571 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:47.510500    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b221fe5e-9ba3-4b22-b088-2274244ab31f","Type":"ContainerStarted","Data":"01d97e1a60b129f10823dd1431252433eac3b73de19cc02f2545ea74750e7f54"}
Jul 28 17:12:47.517507 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:47.517215    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:12:48.516125 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:48.516094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b221fe5e-9ba3-4b22-b088-2274244ab31f","Type":"ContainerStarted","Data":"58e1d836b0fe27f77aa368e945b6cd0d0851705885e31dc6ee795ea511bc64e2"}
Jul 28 17:12:48.516513 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:48.516157    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b221fe5e-9ba3-4b22-b088-2274244ab31f","Type":"ContainerStarted","Data":"b9861df369c4249cd14308b8831b18787fd59d85cc84e484a8e877d2846dd763"}
Jul 28 17:12:48.516513 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:48.516189    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b221fe5e-9ba3-4b22-b088-2274244ab31f","Type":"ContainerStarted","Data":"c03a3486b9271783e5fc9091b2fd81bc8c68cd8f38976d777d7f158ceaa28c4f"}
Jul 28 17:12:48.549339 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:48.549295    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.65603479 podStartE2EDuration="6.549279528s" totalImagesPullingTime="3.878652348s" totalInitContainerRuntime="14.59239ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:48.54651994 +0000 UTC m=+127.190489430" watchObservedRunningTime="2026-07-28 17:12:48.549279528 +0000 UTC m=+127.193249017"
Jul 28 17:12:49.495436 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:49.495408    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5d4cbb8df6-2ljld"
Jul 28 17:12:51.542839 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:51.542802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:12:51.545126 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:51.545102    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9cdd97a5-b1fd-4756-9933-1a551767c9c6-metrics-certs\") pod \"network-metrics-daemon-6czkd\" (UID: \"9cdd97a5-b1fd-4756-9933-1a551767c9c6\") " pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:12:51.818902 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:51.818829    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sw5vg\""
Jul 28 17:12:51.825572 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:51.825549    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6czkd"
Jul 28 17:12:52.058289 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:52.058265    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6czkd"]
Jul 28 17:12:52.060935 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:12:52.060897    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdd97a5_b1fd_4756_9933_1a551767c9c6.slice/crio-8de0ab1133bb84f03b4e81dfcea4aed6bf54ce0e6d899cef91d8ad9ad7d464fd WatchSource:0}: Error finding container 8de0ab1133bb84f03b4e81dfcea4aed6bf54ce0e6d899cef91d8ad9ad7d464fd: Status 404 returned error can't find the container with id 8de0ab1133bb84f03b4e81dfcea4aed6bf54ce0e6d899cef91d8ad9ad7d464fd
Jul 28 17:12:52.533026 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:52.532992    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6czkd" event={"ID":"9cdd97a5-b1fd-4756-9933-1a551767c9c6","Type":"ContainerStarted","Data":"8de0ab1133bb84f03b4e81dfcea4aed6bf54ce0e6d899cef91d8ad9ad7d464fd"}
Jul 28 17:12:52.807112 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:52.807040    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:53.641718 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:53.641677    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b76b7f848-9rz5g"]
Jul 28 17:12:54.543176 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:54.543139    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6czkd" event={"ID":"9cdd97a5-b1fd-4756-9933-1a551767c9c6","Type":"ContainerStarted","Data":"b7893cdcbcf952de36a3b90c8a7a21ad825a23a7bc1c707727c62321113e0014"}
Jul 28 17:12:54.543627 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:54.543194    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6czkd" event={"ID":"9cdd97a5-b1fd-4756-9933-1a551767c9c6","Type":"ContainerStarted","Data":"a0c684a3b0767826c02618dbf4d45850740e4bea5eefeb62437e6be90fcd72ab"}
Jul 28 17:12:54.562481 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:12:54.562431    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6czkd" podStartSLOduration=130.888210218 podStartE2EDuration="2m12.562415416s" totalImagesPullingTime="1.674205198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:54.561413262 +0000 UTC m=+133.205382752" watchObservedRunningTime="2026-07-28 17:12:54.562415416 +0000 UTC m=+133.206384907"
Jul 28 17:13:18.661284 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:18.661218    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b76b7f848-9rz5g" podUID="96a5c68d-f9b9-42c3-b08f-17e90a456cc1" containerName="console" containerID="cri-o://dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483" gracePeriod=15
Jul 28 17:13:18.903695 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:18.903670    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:13:19.076126 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076042    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-oauth-serving-cert\") pod \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") "
Jul 28 17:13:19.076126 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076109    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-oauth-config\") pod \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") "
Jul 28 17:13:19.076417 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076140    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8mqq\" (UniqueName: \"kubernetes.io/projected/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-kube-api-access-c8mqq\") pod \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") "
Jul 28 17:13:19.076417 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076280    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-serving-cert\") pod \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") "
Jul 28 17:13:19.076417 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076332    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-config\") pod \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") "
Jul 28 17:13:19.076417 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076365    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-service-ca\") pod \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") "
Jul 28 17:13:19.076613 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076442    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-trusted-ca-bundle\") pod \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\" (UID: \"96a5c68d-f9b9-42c3-b08f-17e90a456cc1\") "
Jul 28 17:13:19.076613 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076437    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "96a5c68d-f9b9-42c3-b08f-17e90a456cc1" (UID: "96a5c68d-f9b9-42c3-b08f-17e90a456cc1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:19.076741 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076725    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-oauth-serving-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:13:19.076796 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076719    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-config" (OuterVolumeSpecName: "console-config") pod "96a5c68d-f9b9-42c3-b08f-17e90a456cc1" (UID: "96a5c68d-f9b9-42c3-b08f-17e90a456cc1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:19.076853 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-service-ca" (OuterVolumeSpecName: "service-ca") pod "96a5c68d-f9b9-42c3-b08f-17e90a456cc1" (UID: "96a5c68d-f9b9-42c3-b08f-17e90a456cc1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:19.076949 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.076930    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "96a5c68d-f9b9-42c3-b08f-17e90a456cc1" (UID: "96a5c68d-f9b9-42c3-b08f-17e90a456cc1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:13:19.078516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.078487    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-kube-api-access-c8mqq" (OuterVolumeSpecName: "kube-api-access-c8mqq") pod "96a5c68d-f9b9-42c3-b08f-17e90a456cc1" (UID: "96a5c68d-f9b9-42c3-b08f-17e90a456cc1"). InnerVolumeSpecName "kube-api-access-c8mqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:13:19.078516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.078510    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "96a5c68d-f9b9-42c3-b08f-17e90a456cc1" (UID: "96a5c68d-f9b9-42c3-b08f-17e90a456cc1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:19.078652 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.078532    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "96a5c68d-f9b9-42c3-b08f-17e90a456cc1" (UID: "96a5c68d-f9b9-42c3-b08f-17e90a456cc1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:13:19.177317 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.177293    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-oauth-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:13:19.177317 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.177314    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8mqq\" (UniqueName: \"kubernetes.io/projected/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-kube-api-access-c8mqq\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:13:19.177443 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.177323    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-serving-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:13:19.177443 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.177332    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-console-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:13:19.177443 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.177341    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-service-ca\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:13:19.177443 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.177349    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/96a5c68d-f9b9-42c3-b08f-17e90a456cc1-trusted-ca-bundle\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:13:19.646018 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.645984    2577 generic.go:356] "Generic (PLEG): container finished" podID="96a5c68d-f9b9-42c3-b08f-17e90a456cc1" containerID="dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483" exitCode=0
Jul 28 17:13:19.646185 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.646027    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b76b7f848-9rz5g"
Jul 28 17:13:19.646185 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.646072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b76b7f848-9rz5g" event={"ID":"96a5c68d-f9b9-42c3-b08f-17e90a456cc1","Type":"ContainerDied","Data":"dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483"}
Jul 28 17:13:19.646185 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.646121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b76b7f848-9rz5g" event={"ID":"96a5c68d-f9b9-42c3-b08f-17e90a456cc1","Type":"ContainerDied","Data":"4ce63e93f54ccab35191d2af6c1f9b8631c17a3487c8778467e7719e030bb0a3"}
Jul 28 17:13:19.646185 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.646144    2577 scope.go:117] "RemoveContainer" containerID="dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483"
Jul 28 17:13:19.654594 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.654576    2577 scope.go:117] "RemoveContainer" containerID="dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483"
Jul 28 17:13:19.654863 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:13:19.654844    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483\": container with ID starting with dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483 not found: ID does not exist" containerID="dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483"
Jul 28 17:13:19.654911 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.654871    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483"} err="failed to get container status \"dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483\": rpc error: code = NotFound desc = could not find container \"dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483\": container with ID starting with dddb6d40928746f20ec58767c74db7b6e4ed98d273135838e5201b8942be5483 not found: ID does not exist"
Jul 28 17:13:19.666664 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.666643    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b76b7f848-9rz5g"]
Jul 28 17:13:19.672630 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.672612    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b76b7f848-9rz5g"]
Jul 28 17:13:19.909266 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:19.909170    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a5c68d-f9b9-42c3-b08f-17e90a456cc1" path="/var/lib/kubelet/pods/96a5c68d-f9b9-42c3-b08f-17e90a456cc1/volumes"
Jul 28 17:13:42.807011 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:42.806976    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:42.822730 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:42.822701    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:43.752534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:13:43.752507    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:15:41.732008 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:15:41.731976    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:15:41.737141 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:15:41.737117    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:15:41.743339 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:15:41.743305    2577 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:18:57.104627 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.104595    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-snxwh"]
Jul 28 17:18:57.105060 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.104965    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a5c68d-f9b9-42c3-b08f-17e90a456cc1" containerName="console"
Jul 28 17:18:57.105060 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.104978    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a5c68d-f9b9-42c3-b08f-17e90a456cc1" containerName="console"
Jul 28 17:18:57.105060 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.105044    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a5c68d-f9b9-42c3-b08f-17e90a456cc1" containerName="console"
Jul 28 17:18:57.107864 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.107849    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-snxwh"
Jul 28 17:18:57.110430 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.110411    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hjb5m\""
Jul 28 17:18:57.110604 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.110588    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:18:57.111614 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.111596    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:18:57.111668 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.111631    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:18:57.116440 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.116420    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-snxwh"]
Jul 28 17:18:57.166958 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.166932    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2584fe5c-f355-4eb8-93b5-64ff9ed74149-model-cache\") pod \"s3-init-snxwh\" (UID: \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\") " pod="kserve/s3-init-snxwh"
Jul 28 17:18:57.167051 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.166967    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcvng\" (UniqueName: \"kubernetes.io/projected/2584fe5c-f355-4eb8-93b5-64ff9ed74149-kube-api-access-bcvng\") pod \"s3-init-snxwh\" (UID: \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\") " pod="kserve/s3-init-snxwh"
Jul 28 17:18:57.268175 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.268151    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2584fe5c-f355-4eb8-93b5-64ff9ed74149-model-cache\") pod \"s3-init-snxwh\" (UID: \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\") " pod="kserve/s3-init-snxwh"
Jul 28 17:18:57.268290 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.268183    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcvng\" (UniqueName: \"kubernetes.io/projected/2584fe5c-f355-4eb8-93b5-64ff9ed74149-kube-api-access-bcvng\") pod \"s3-init-snxwh\" (UID: \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\") " pod="kserve/s3-init-snxwh"
Jul 28 17:18:57.268538 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.268518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2584fe5c-f355-4eb8-93b5-64ff9ed74149-model-cache\") pod \"s3-init-snxwh\" (UID: \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\") " pod="kserve/s3-init-snxwh"
Jul 28 17:18:57.277068 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.277049    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcvng\" (UniqueName: \"kubernetes.io/projected/2584fe5c-f355-4eb8-93b5-64ff9ed74149-kube-api-access-bcvng\") pod \"s3-init-snxwh\" (UID: \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\") " pod="kserve/s3-init-snxwh"
Jul 28 17:18:57.430000 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.429975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-snxwh"
Jul 28 17:18:57.554729 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.554706    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-snxwh"]
Jul 28 17:18:57.557431 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:18:57.557407    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2584fe5c_f355_4eb8_93b5_64ff9ed74149.slice/crio-6808f74d27efabcc075ec7228e934be6478c9448a4803fe1cc6aa010397ccc70 WatchSource:0}: Error finding container 6808f74d27efabcc075ec7228e934be6478c9448a4803fe1cc6aa010397ccc70: Status 404 returned error can't find the container with id 6808f74d27efabcc075ec7228e934be6478c9448a4803fe1cc6aa010397ccc70
Jul 28 17:18:57.559670 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.559656    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:18:57.891194 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:18:57.891159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-snxwh" event={"ID":"2584fe5c-f355-4eb8-93b5-64ff9ed74149","Type":"ContainerStarted","Data":"6808f74d27efabcc075ec7228e934be6478c9448a4803fe1cc6aa010397ccc70"}
Jul 28 17:19:00.904683 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:19:00.904646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-snxwh" event={"ID":"2584fe5c-f355-4eb8-93b5-64ff9ed74149","Type":"ContainerStarted","Data":"c75ccf8a6fc1628a4b0f0a8d8b99a1090be93c064cf18354fd9e8b1d93ee8820"}
Jul 28 17:19:05.924660 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:19:05.924582    2577 generic.go:356] "Generic (PLEG): container finished" podID="2584fe5c-f355-4eb8-93b5-64ff9ed74149" containerID="c75ccf8a6fc1628a4b0f0a8d8b99a1090be93c064cf18354fd9e8b1d93ee8820" exitCode=0
Jul 28 17:19:05.924660 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:19:05.924628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-snxwh" event={"ID":"2584fe5c-f355-4eb8-93b5-64ff9ed74149","Type":"ContainerDied","Data":"c75ccf8a6fc1628a4b0f0a8d8b99a1090be93c064cf18354fd9e8b1d93ee8820"}
Jul 28 17:19:10.949291 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:19:10.949261    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-snxwh" event={"ID":"2584fe5c-f355-4eb8-93b5-64ff9ed74149","Type":"ContainerStarted","Data":"0ffa871959914d1b1ed098938e629158b258d6a45f973620eacfe9f6997742a0"}
Jul 28 17:19:10.968599 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:19:10.968551    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-snxwh" podStartSLOduration=1.397817008 podStartE2EDuration="13.96853713s" totalImagesPullingTime="7.657835565s" totalInitContainerRuntime="4.912884557s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:10.9663568 +0000 UTC m=+509.610326289" watchObservedRunningTime="2026-07-28 17:19:10.96853713 +0000 UTC m=+509.612506619"
Jul 28 17:20:16.196886 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:16.196853    2577 generic.go:356] "Generic (PLEG): container finished" podID="2584fe5c-f355-4eb8-93b5-64ff9ed74149" containerID="0ffa871959914d1b1ed098938e629158b258d6a45f973620eacfe9f6997742a0" exitCode=0
Jul 28 17:20:16.197400 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:16.196928    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-snxwh" event={"ID":"2584fe5c-f355-4eb8-93b5-64ff9ed74149","Type":"ContainerDied","Data":"0ffa871959914d1b1ed098938e629158b258d6a45f973620eacfe9f6997742a0"}
Jul 28 17:20:17.323668 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:17.323646    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-snxwh"
Jul 28 17:20:17.387794 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:17.387766    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2584fe5c-f355-4eb8-93b5-64ff9ed74149-model-cache\") pod \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\" (UID: \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\") "
Jul 28 17:20:17.387942 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:17.387811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcvng\" (UniqueName: \"kubernetes.io/projected/2584fe5c-f355-4eb8-93b5-64ff9ed74149-kube-api-access-bcvng\") pod \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\" (UID: \"2584fe5c-f355-4eb8-93b5-64ff9ed74149\") "
Jul 28 17:20:17.389995 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:17.389960    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2584fe5c-f355-4eb8-93b5-64ff9ed74149-kube-api-access-bcvng" (OuterVolumeSpecName: "kube-api-access-bcvng") pod "2584fe5c-f355-4eb8-93b5-64ff9ed74149" (UID: "2584fe5c-f355-4eb8-93b5-64ff9ed74149"). InnerVolumeSpecName "kube-api-access-bcvng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:17.406527 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:17.406491    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2584fe5c-f355-4eb8-93b5-64ff9ed74149-model-cache" (OuterVolumeSpecName: "model-cache") pod "2584fe5c-f355-4eb8-93b5-64ff9ed74149" (UID: "2584fe5c-f355-4eb8-93b5-64ff9ed74149"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:20:17.489444 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:17.489375    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2584fe5c-f355-4eb8-93b5-64ff9ed74149-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:20:17.489444 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:17.489401    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcvng\" (UniqueName: \"kubernetes.io/projected/2584fe5c-f355-4eb8-93b5-64ff9ed74149-kube-api-access-bcvng\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:20:18.205599 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:18.205522    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-snxwh"
Jul 28 17:20:18.205755 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:18.205520    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-snxwh" event={"ID":"2584fe5c-f355-4eb8-93b5-64ff9ed74149","Type":"ContainerDied","Data":"6808f74d27efabcc075ec7228e934be6478c9448a4803fe1cc6aa010397ccc70"}
Jul 28 17:20:18.205755 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:18.205635    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6808f74d27efabcc075ec7228e934be6478c9448a4803fe1cc6aa010397ccc70"
Jul 28 17:20:28.038833 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.038798    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-7lhbf"]
Jul 28 17:20:28.039297 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.039092    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2584fe5c-f355-4eb8-93b5-64ff9ed74149" containerName="s3-init"
Jul 28 17:20:28.039297 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.039102    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2584fe5c-f355-4eb8-93b5-64ff9ed74149" containerName="s3-init"
Jul 28 17:20:28.039297 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.039114    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2584fe5c-f355-4eb8-93b5-64ff9ed74149" containerName="download-hf-model"
Jul 28 17:20:28.039297 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.039121    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2584fe5c-f355-4eb8-93b5-64ff9ed74149" containerName="download-hf-model"
Jul 28 17:20:28.039297 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.039181    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2584fe5c-f355-4eb8-93b5-64ff9ed74149" containerName="s3-init"
Jul 28 17:20:28.041287 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.041271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:28.044605 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.044585    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 28 17:20:28.046090 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.046070    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:20:28.046189 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.046068    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hjb5m\""
Jul 28 17:20:28.046189 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.046122    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:20:28.060715 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.060695    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7lhbf"]
Jul 28 17:20:28.184368 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.184337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-model-cache\") pod \"s3-tls-init-custom-7lhbf\" (UID: \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\") " pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:28.184524 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.184414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5klmg\" (UniqueName: \"kubernetes.io/projected/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-kube-api-access-5klmg\") pod \"s3-tls-init-custom-7lhbf\" (UID: \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\") " pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:28.284753 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.284723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5klmg\" (UniqueName: \"kubernetes.io/projected/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-kube-api-access-5klmg\") pod \"s3-tls-init-custom-7lhbf\" (UID: \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\") " pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:28.284869 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.284759    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-model-cache\") pod \"s3-tls-init-custom-7lhbf\" (UID: \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\") " pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:28.285177 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.285160    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-model-cache\") pod \"s3-tls-init-custom-7lhbf\" (UID: \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\") " pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:28.294783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.294730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5klmg\" (UniqueName: \"kubernetes.io/projected/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-kube-api-access-5klmg\") pod \"s3-tls-init-custom-7lhbf\" (UID: \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\") " pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:28.360166 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.360143    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:28.489188 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:28.489052    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-7lhbf"]
Jul 28 17:20:28.491761 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:20:28.491728    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod575d67d0_f00c_4b74_a316_99a8d0c8f4e5.slice/crio-23f98fd12f42bbfccab57b0b15aa10885c4f1bdd0884f46175c20dfe94e9c29d WatchSource:0}: Error finding container 23f98fd12f42bbfccab57b0b15aa10885c4f1bdd0884f46175c20dfe94e9c29d: Status 404 returned error can't find the container with id 23f98fd12f42bbfccab57b0b15aa10885c4f1bdd0884f46175c20dfe94e9c29d
Jul 28 17:20:29.248164 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:29.248128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7lhbf" event={"ID":"575d67d0-f00c-4b74-a316-99a8d0c8f4e5","Type":"ContainerStarted","Data":"50b4986eb3ed4949ae4772862216b4ea56fa20d3e7e0e29c63aa290d3c194c49"}
Jul 28 17:20:29.248543 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:29.248173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7lhbf" event={"ID":"575d67d0-f00c-4b74-a316-99a8d0c8f4e5","Type":"ContainerStarted","Data":"23f98fd12f42bbfccab57b0b15aa10885c4f1bdd0884f46175c20dfe94e9c29d"}
Jul 28 17:20:35.271708 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:35.271674    2577 generic.go:356] "Generic (PLEG): container finished" podID="575d67d0-f00c-4b74-a316-99a8d0c8f4e5" containerID="50b4986eb3ed4949ae4772862216b4ea56fa20d3e7e0e29c63aa290d3c194c49" exitCode=0
Jul 28 17:20:35.272010 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:35.271736    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7lhbf" event={"ID":"575d67d0-f00c-4b74-a316-99a8d0c8f4e5","Type":"ContainerDied","Data":"50b4986eb3ed4949ae4772862216b4ea56fa20d3e7e0e29c63aa290d3c194c49"}
Jul 28 17:20:36.277780 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:36.277748    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7lhbf" event={"ID":"575d67d0-f00c-4b74-a316-99a8d0c8f4e5","Type":"ContainerStarted","Data":"399b66b017d39567b6fba1dfb2b6628709c88cb3d23c8d8c37414a03d714e36f"}
Jul 28 17:20:36.295784 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:36.295737    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-7lhbf" podStartSLOduration=1.643394552 podStartE2EDuration="8.295722511s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.652327959s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:36.294673849 +0000 UTC m=+594.938643316" watchObservedRunningTime="2026-07-28 17:20:36.295722511 +0000 UTC m=+594.939692001"
Jul 28 17:20:41.765664 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:41.765630    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:20:41.768644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:41.768620    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:20:42.300265 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:42.300222    2577 generic.go:356] "Generic (PLEG): container finished" podID="575d67d0-f00c-4b74-a316-99a8d0c8f4e5" containerID="399b66b017d39567b6fba1dfb2b6628709c88cb3d23c8d8c37414a03d714e36f" exitCode=0
Jul 28 17:20:42.300388 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:42.300281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7lhbf" event={"ID":"575d67d0-f00c-4b74-a316-99a8d0c8f4e5","Type":"ContainerDied","Data":"399b66b017d39567b6fba1dfb2b6628709c88cb3d23c8d8c37414a03d714e36f"}
Jul 28 17:20:43.443093 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:43.443067    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:43.607102 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:43.607037    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-model-cache\") pod \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\" (UID: \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\") "
Jul 28 17:20:43.607102 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:43.607090    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5klmg\" (UniqueName: \"kubernetes.io/projected/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-kube-api-access-5klmg\") pod \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\" (UID: \"575d67d0-f00c-4b74-a316-99a8d0c8f4e5\") "
Jul 28 17:20:43.609045 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:43.609018    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-kube-api-access-5klmg" (OuterVolumeSpecName: "kube-api-access-5klmg") pod "575d67d0-f00c-4b74-a316-99a8d0c8f4e5" (UID: "575d67d0-f00c-4b74-a316-99a8d0c8f4e5"). InnerVolumeSpecName "kube-api-access-5klmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:43.625410 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:43.625386    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "575d67d0-f00c-4b74-a316-99a8d0c8f4e5" (UID: "575d67d0-f00c-4b74-a316-99a8d0c8f4e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:20:43.708478 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:43.708455    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5klmg\" (UniqueName: \"kubernetes.io/projected/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-kube-api-access-5klmg\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:20:43.708478 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:43.708475    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/575d67d0-f00c-4b74-a316-99a8d0c8f4e5-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:20:44.308553 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:44.308473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-7lhbf" event={"ID":"575d67d0-f00c-4b74-a316-99a8d0c8f4e5","Type":"ContainerDied","Data":"23f98fd12f42bbfccab57b0b15aa10885c4f1bdd0884f46175c20dfe94e9c29d"}
Jul 28 17:20:44.308553 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:44.308502    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-7lhbf"
Jul 28 17:20:44.308553 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:44.308509    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23f98fd12f42bbfccab57b0b15aa10885c4f1bdd0884f46175c20dfe94e9c29d"
Jul 28 17:20:47.396646 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.396615    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-sfwbj"]
Jul 28 17:20:47.397030 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.396914    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575d67d0-f00c-4b74-a316-99a8d0c8f4e5" containerName="s3-tls-init-custom"
Jul 28 17:20:47.397030 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.396924    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="575d67d0-f00c-4b74-a316-99a8d0c8f4e5" containerName="s3-tls-init-custom"
Jul 28 17:20:47.397030 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.396947    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="575d67d0-f00c-4b74-a316-99a8d0c8f4e5" containerName="download-hf-model"
Jul 28 17:20:47.397030 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.396952    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="575d67d0-f00c-4b74-a316-99a8d0c8f4e5" containerName="download-hf-model"
Jul 28 17:20:47.397030 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.396997    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="575d67d0-f00c-4b74-a316-99a8d0c8f4e5" containerName="s3-tls-init-custom"
Jul 28 17:20:47.398990 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.398973    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:20:47.402291 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.402269    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hjb5m\""
Jul 28 17:20:47.402291 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.402281    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:20:47.403187 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.403173    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 28 17:20:47.403269 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.403196    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:20:47.409297 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.409278    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-sfwbj"]
Jul 28 17:20:47.437278 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.437255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710814b1-2021-4845-96b8-6352b64b667c-model-cache\") pod \"s3-tls-init-serving-sfwbj\" (UID: \"710814b1-2021-4845-96b8-6352b64b667c\") " pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:20:47.437387 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.437283    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqgh5\" (UniqueName: \"kubernetes.io/projected/710814b1-2021-4845-96b8-6352b64b667c-kube-api-access-cqgh5\") pod \"s3-tls-init-serving-sfwbj\" (UID: \"710814b1-2021-4845-96b8-6352b64b667c\") " pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:20:47.537653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.537631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710814b1-2021-4845-96b8-6352b64b667c-model-cache\") pod \"s3-tls-init-serving-sfwbj\" (UID: \"710814b1-2021-4845-96b8-6352b64b667c\") " pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:20:47.537746 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.537656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqgh5\" (UniqueName: \"kubernetes.io/projected/710814b1-2021-4845-96b8-6352b64b667c-kube-api-access-cqgh5\") pod \"s3-tls-init-serving-sfwbj\" (UID: \"710814b1-2021-4845-96b8-6352b64b667c\") " pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:20:47.537961 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.537944    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710814b1-2021-4845-96b8-6352b64b667c-model-cache\") pod \"s3-tls-init-serving-sfwbj\" (UID: \"710814b1-2021-4845-96b8-6352b64b667c\") " pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:20:47.547116 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.547093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqgh5\" (UniqueName: \"kubernetes.io/projected/710814b1-2021-4845-96b8-6352b64b667c-kube-api-access-cqgh5\") pod \"s3-tls-init-serving-sfwbj\" (UID: \"710814b1-2021-4845-96b8-6352b64b667c\") " pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:20:47.716661 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.716587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:20:47.839806 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:47.839785    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-sfwbj"]
Jul 28 17:20:47.841548 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:20:47.841513    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod710814b1_2021_4845_96b8_6352b64b667c.slice/crio-55c3416ce3c6cdd3aa69097f0b3e88fb87e7c2455b6c9b0b94f6b8aae3d0cc43 WatchSource:0}: Error finding container 55c3416ce3c6cdd3aa69097f0b3e88fb87e7c2455b6c9b0b94f6b8aae3d0cc43: Status 404 returned error can't find the container with id 55c3416ce3c6cdd3aa69097f0b3e88fb87e7c2455b6c9b0b94f6b8aae3d0cc43
Jul 28 17:20:48.326439 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:48.326401    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-sfwbj" event={"ID":"710814b1-2021-4845-96b8-6352b64b667c","Type":"ContainerStarted","Data":"f4de0a7dfbbbac8dd33fe31921cde2ce6b0a0f76a45bda656098457e612872f3"}
Jul 28 17:20:48.326587 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:48.326447    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-sfwbj" event={"ID":"710814b1-2021-4845-96b8-6352b64b667c","Type":"ContainerStarted","Data":"55c3416ce3c6cdd3aa69097f0b3e88fb87e7c2455b6c9b0b94f6b8aae3d0cc43"}
Jul 28 17:20:54.351532 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:54.351498    2577 generic.go:356] "Generic (PLEG): container finished" podID="710814b1-2021-4845-96b8-6352b64b667c" containerID="f4de0a7dfbbbac8dd33fe31921cde2ce6b0a0f76a45bda656098457e612872f3" exitCode=0
Jul 28 17:20:54.351849 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:54.351557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-sfwbj" event={"ID":"710814b1-2021-4845-96b8-6352b64b667c","Type":"ContainerDied","Data":"f4de0a7dfbbbac8dd33fe31921cde2ce6b0a0f76a45bda656098457e612872f3"}
Jul 28 17:20:55.356933 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:55.356899    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-sfwbj" event={"ID":"710814b1-2021-4845-96b8-6352b64b667c","Type":"ContainerStarted","Data":"62ab6f3c6c7acc3337ad66d3ca83df7e1200a9cf7d3b1f4070410ae41739b4ee"}
Jul 28 17:20:55.375863 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:20:55.375821    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-sfwbj" podStartSLOduration=2.163336935 podStartE2EDuration="8.375808602s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.212471667s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:55.37414105 +0000 UTC m=+614.018110558" watchObservedRunningTime="2026-07-28 17:20:55.375808602 +0000 UTC m=+614.019778091"
Jul 28 17:21:00.378764 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:00.378728    2577 generic.go:356] "Generic (PLEG): container finished" podID="710814b1-2021-4845-96b8-6352b64b667c" containerID="62ab6f3c6c7acc3337ad66d3ca83df7e1200a9cf7d3b1f4070410ae41739b4ee" exitCode=0
Jul 28 17:21:00.379312 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:00.378815    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-sfwbj" event={"ID":"710814b1-2021-4845-96b8-6352b64b667c","Type":"ContainerDied","Data":"62ab6f3c6c7acc3337ad66d3ca83df7e1200a9cf7d3b1f4070410ae41739b4ee"}
Jul 28 17:21:01.529418 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:01.529394    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:21:01.640913 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:01.640890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710814b1-2021-4845-96b8-6352b64b667c-model-cache\") pod \"710814b1-2021-4845-96b8-6352b64b667c\" (UID: \"710814b1-2021-4845-96b8-6352b64b667c\") "
Jul 28 17:21:01.641038 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:01.640951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqgh5\" (UniqueName: \"kubernetes.io/projected/710814b1-2021-4845-96b8-6352b64b667c-kube-api-access-cqgh5\") pod \"710814b1-2021-4845-96b8-6352b64b667c\" (UID: \"710814b1-2021-4845-96b8-6352b64b667c\") "
Jul 28 17:21:01.642919 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:01.642897    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/710814b1-2021-4845-96b8-6352b64b667c-kube-api-access-cqgh5" (OuterVolumeSpecName: "kube-api-access-cqgh5") pod "710814b1-2021-4845-96b8-6352b64b667c" (UID: "710814b1-2021-4845-96b8-6352b64b667c"). InnerVolumeSpecName "kube-api-access-cqgh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:21:01.660408 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:01.660384    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/710814b1-2021-4845-96b8-6352b64b667c-model-cache" (OuterVolumeSpecName: "model-cache") pod "710814b1-2021-4845-96b8-6352b64b667c" (UID: "710814b1-2021-4845-96b8-6352b64b667c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:21:01.741872 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:01.741850    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/710814b1-2021-4845-96b8-6352b64b667c-model-cache\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:21:01.741872 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:01.741869    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqgh5\" (UniqueName: \"kubernetes.io/projected/710814b1-2021-4845-96b8-6352b64b667c-kube-api-access-cqgh5\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:21:02.389043 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:02.389012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-sfwbj" event={"ID":"710814b1-2021-4845-96b8-6352b64b667c","Type":"ContainerDied","Data":"55c3416ce3c6cdd3aa69097f0b3e88fb87e7c2455b6c9b0b94f6b8aae3d0cc43"}
Jul 28 17:21:02.389287 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:02.389044    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-sfwbj"
Jul 28 17:21:02.389287 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:21:02.389051    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c3416ce3c6cdd3aa69097f0b3e88fb87e7c2455b6c9b0b94f6b8aae3d0cc43"
Jul 28 17:25:41.797148 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:25:41.797069    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:25:41.797698 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:25:41.797573    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:30:41.829467 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:30:41.829442    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:30:41.829467 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:30:41.829454    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:35:41.855197 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:35:41.855167    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:35:41.856654 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:35:41.856629    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:39:57.216113 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.216078    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"]
Jul 28 17:39:57.216552 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.216536    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710814b1-2021-4845-96b8-6352b64b667c" containerName="s3-tls-init-serving"
Jul 28 17:39:57.216552 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.216553    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="710814b1-2021-4845-96b8-6352b64b667c" containerName="s3-tls-init-serving"
Jul 28 17:39:57.216631 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.216563    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="710814b1-2021-4845-96b8-6352b64b667c" containerName="download-hf-model"
Jul 28 17:39:57.216631 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.216569    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="710814b1-2021-4845-96b8-6352b64b667c" containerName="download-hf-model"
Jul 28 17:39:57.216631 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.216630    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="710814b1-2021-4845-96b8-6352b64b667c" containerName="s3-tls-init-serving"
Jul 28 17:39:57.219628 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.219612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.222395 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.222373    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 17:39:57.222395 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.222385    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 17:39:57.222564 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.222413    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:39:57.222839 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.222823    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:39:57.224051 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.224031    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:39:57.224134 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.224088    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:39:57.224134 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.224121    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:39:57.231500 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.231478    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"]
Jul 28 17:39:57.239141 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.239121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041e9779-cae3-4fac-887d-a6f0adf14247-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.239231 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.239153    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/041e9779-cae3-4fac-887d-a6f0adf14247-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.239231 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.239174    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbc4p\" (UniqueName: \"kubernetes.io/projected/041e9779-cae3-4fac-887d-a6f0adf14247-kube-api-access-wbc4p\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.239231 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.239193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.239378 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.239272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.339969 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.339933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbc4p\" (UniqueName: \"kubernetes.io/projected/041e9779-cae3-4fac-887d-a6f0adf14247-kube-api-access-wbc4p\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.339969 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.339972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.340217 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.340076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.340217 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.340129    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041e9779-cae3-4fac-887d-a6f0adf14247-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.340217 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.340152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/041e9779-cae3-4fac-887d-a6f0adf14247-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.340639 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.340616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041e9779-cae3-4fac-887d-a6f0adf14247-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.340825 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.340807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.340903 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.340832    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.342662 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.342643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/041e9779-cae3-4fac-887d-a6f0adf14247-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.349832 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.349810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbc4p\" (UniqueName: \"kubernetes.io/projected/041e9779-cae3-4fac-887d-a6f0adf14247-kube-api-access-wbc4p\") pod \"isvc-canary-promote-v2-predictor-f69c9c758-bq94j\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.530011 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.529924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:39:57.661957 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.661932    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"]
Jul 28 17:39:57.664482 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:39:57.664452    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041e9779_cae3_4fac_887d_a6f0adf14247.slice/crio-0e6435d07da93c8448b576c2537e5fef1c9ec41add7ee5fcf50607fc550d7415 WatchSource:0}: Error finding container 0e6435d07da93c8448b576c2537e5fef1c9ec41add7ee5fcf50607fc550d7415: Status 404 returned error can't find the container with id 0e6435d07da93c8448b576c2537e5fef1c9ec41add7ee5fcf50607fc550d7415
Jul 28 17:39:57.801296 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.801193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" event={"ID":"041e9779-cae3-4fac-887d-a6f0adf14247","Type":"ContainerStarted","Data":"6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a"}
Jul 28 17:39:57.801296 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:39:57.801280    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" event={"ID":"041e9779-cae3-4fac-887d-a6f0adf14247","Type":"ContainerStarted","Data":"0e6435d07da93c8448b576c2537e5fef1c9ec41add7ee5fcf50607fc550d7415"}
Jul 28 17:40:01.817138 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:01.817048    2577 generic.go:356] "Generic (PLEG): container finished" podID="041e9779-cae3-4fac-887d-a6f0adf14247" containerID="6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a" exitCode=0
Jul 28 17:40:01.817548 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:01.817128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" event={"ID":"041e9779-cae3-4fac-887d-a6f0adf14247","Type":"ContainerDied","Data":"6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a"}
Jul 28 17:40:01.818223 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:01.818209    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:40:14.887657 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:14.887626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" event={"ID":"041e9779-cae3-4fac-887d-a6f0adf14247","Type":"ContainerStarted","Data":"95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1"}
Jul 28 17:40:17.905340 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:17.905299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" event={"ID":"041e9779-cae3-4fac-887d-a6f0adf14247","Type":"ContainerStarted","Data":"6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1"}
Jul 28 17:40:17.905746 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:17.905383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:40:17.927055 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:17.926989    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podStartSLOduration=1.9931731099999999 podStartE2EDuration="20.926971234s" totalImagesPullingTime="15.145970042s" totalInitContainerRuntime="3.787828082s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:17.926126276 +0000 UTC m=+1776.570095766" watchObservedRunningTime="2026-07-28 17:40:17.926971234 +0000 UTC m=+1776.570940724"
Jul 28 17:40:18.909613 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:18.909581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:40:18.910720 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:18.910687    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:40:19.912477 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:19.912436    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:40:24.917409 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:24.917384    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:40:24.917867 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:24.917840    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:40:34.918301 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:34.918261    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:40:41.883644 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:41.883609    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:40:41.884169 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:41.883718    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:40:44.918741 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:44.918698    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:40:54.917876 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:40:54.917836    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:41:04.918199 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:04.918159    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:41:14.918640 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:14.918601    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:41:24.918394 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:24.918352    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:41:31.904396 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:31.904369    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:41:37.462794 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:37.462749    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"]
Jul 28 17:41:37.463338 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:37.462983    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container" containerID="cri-o://95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1" gracePeriod=30
Jul 28 17:41:37.463338 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:37.463024    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kube-rbac-proxy" containerID="cri-o://6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1" gracePeriod=30
Jul 28 17:41:38.225635 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:38.225603    2577 generic.go:356] "Generic (PLEG): container finished" podID="041e9779-cae3-4fac-887d-a6f0adf14247" containerID="6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1" exitCode=2
Jul 28 17:41:38.225808 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:38.225680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" event={"ID":"041e9779-cae3-4fac-887d-a6f0adf14247","Type":"ContainerDied","Data":"6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1"}
Jul 28 17:41:39.913069 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:39.913028    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 28 17:41:41.902191 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:41.902166    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:41:41.999822 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:41.999744    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/041e9779-cae3-4fac-887d-a6f0adf14247-proxy-tls\") pod \"041e9779-cae3-4fac-887d-a6f0adf14247\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") "
Jul 28 17:41:41.999822 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:41.999804    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-cabundle-cert\") pod \"041e9779-cae3-4fac-887d-a6f0adf14247\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") "
Jul 28 17:41:41.999997 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:41.999835    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041e9779-cae3-4fac-887d-a6f0adf14247-kserve-provision-location\") pod \"041e9779-cae3-4fac-887d-a6f0adf14247\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") "
Jul 28 17:41:41.999997 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:41.999869    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbc4p\" (UniqueName: \"kubernetes.io/projected/041e9779-cae3-4fac-887d-a6f0adf14247-kube-api-access-wbc4p\") pod \"041e9779-cae3-4fac-887d-a6f0adf14247\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") "
Jul 28 17:41:41.999997 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:41.999906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-kube-rbac-proxy-sar-config\") pod \"041e9779-cae3-4fac-887d-a6f0adf14247\" (UID: \"041e9779-cae3-4fac-887d-a6f0adf14247\") "
Jul 28 17:41:42.000203 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.000167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041e9779-cae3-4fac-887d-a6f0adf14247-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "041e9779-cae3-4fac-887d-a6f0adf14247" (UID: "041e9779-cae3-4fac-887d-a6f0adf14247"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:41:42.000345 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.000219    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "041e9779-cae3-4fac-887d-a6f0adf14247" (UID: "041e9779-cae3-4fac-887d-a6f0adf14247"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:42.000345 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.000303    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "041e9779-cae3-4fac-887d-a6f0adf14247" (UID: "041e9779-cae3-4fac-887d-a6f0adf14247"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:42.001997 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.001972    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/041e9779-cae3-4fac-887d-a6f0adf14247-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "041e9779-cae3-4fac-887d-a6f0adf14247" (UID: "041e9779-cae3-4fac-887d-a6f0adf14247"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:41:42.002061 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.002030    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041e9779-cae3-4fac-887d-a6f0adf14247-kube-api-access-wbc4p" (OuterVolumeSpecName: "kube-api-access-wbc4p") pod "041e9779-cae3-4fac-887d-a6f0adf14247" (UID: "041e9779-cae3-4fac-887d-a6f0adf14247"). InnerVolumeSpecName "kube-api-access-wbc4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:42.101173 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.101144    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:41:42.101173 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.101168    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/041e9779-cae3-4fac-887d-a6f0adf14247-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:41:42.101173 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.101177    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbc4p\" (UniqueName: \"kubernetes.io/projected/041e9779-cae3-4fac-887d-a6f0adf14247-kube-api-access-wbc4p\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:41:42.101390 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.101185    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/041e9779-cae3-4fac-887d-a6f0adf14247-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:41:42.101390 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.101195    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/041e9779-cae3-4fac-887d-a6f0adf14247-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:41:42.246583 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.246548    2577 generic.go:356] "Generic (PLEG): container finished" podID="041e9779-cae3-4fac-887d-a6f0adf14247" containerID="95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1" exitCode=0
Jul 28 17:41:42.246780 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.246590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" event={"ID":"041e9779-cae3-4fac-887d-a6f0adf14247","Type":"ContainerDied","Data":"95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1"}
Jul 28 17:41:42.246780 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.246646    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"
Jul 28 17:41:42.246780 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.246715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j" event={"ID":"041e9779-cae3-4fac-887d-a6f0adf14247","Type":"ContainerDied","Data":"0e6435d07da93c8448b576c2537e5fef1c9ec41add7ee5fcf50607fc550d7415"}
Jul 28 17:41:42.246780 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.246744    2577 scope.go:117] "RemoveContainer" containerID="6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1"
Jul 28 17:41:42.257598 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.257582    2577 scope.go:117] "RemoveContainer" containerID="95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1"
Jul 28 17:41:42.264536 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.264516    2577 scope.go:117] "RemoveContainer" containerID="6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a"
Jul 28 17:41:42.271515 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.271500    2577 scope.go:117] "RemoveContainer" containerID="6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1"
Jul 28 17:41:42.271780 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:41:42.271761    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1\": container with ID starting with 6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1 not found: ID does not exist" containerID="6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1"
Jul 28 17:41:42.271847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.271788    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1"} err="failed to get container status \"6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1\": rpc error: code = NotFound desc = could not find container \"6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1\": container with ID starting with 6b378fc9022e2011d4bf307c1f0b1ff783fca19b0ba1f5d2a3f4ed0a37a16ea1 not found: ID does not exist"
Jul 28 17:41:42.271847 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.271806    2577 scope.go:117] "RemoveContainer" containerID="95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1"
Jul 28 17:41:42.272051 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:41:42.272022    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1\": container with ID starting with 95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1 not found: ID does not exist" containerID="95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1"
Jul 28 17:41:42.272129 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.272055    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1"} err="failed to get container status \"95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1\": rpc error: code = NotFound desc = could not find container \"95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1\": container with ID starting with 95c49fa8c905520c8774ffcbae1abfd13f3c0b48289864ccaac5c2c77d7426b1 not found: ID does not exist"
Jul 28 17:41:42.272129 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.272073    2577 scope.go:117] "RemoveContainer" containerID="6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a"
Jul 28 17:41:42.272294 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.272279    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"]
Jul 28 17:41:42.272294 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:41:42.272289    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a\": container with ID starting with 6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a not found: ID does not exist" containerID="6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a"
Jul 28 17:41:42.272371 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.272304    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a"} err="failed to get container status \"6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a\": rpc error: code = NotFound desc = could not find container \"6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a\": container with ID starting with 6e61104a9f09f0bf6de23e5936fdc67f2bd37eae59c8522538e07e7d7547799a not found: ID does not exist"
Jul 28 17:41:42.276111 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.276090    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-f69c9c758-bq94j"]
Jul 28 17:41:42.632950 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.632867    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"]
Jul 28 17:41:42.633418 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.633401    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="storage-initializer"
Jul 28 17:41:42.633477 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.633420    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="storage-initializer"
Jul 28 17:41:42.633477 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.633435    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kube-rbac-proxy"
Jul 28 17:41:42.633477 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.633441    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kube-rbac-proxy"
Jul 28 17:41:42.633477 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.633457    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container"
Jul 28 17:41:42.633477 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.633463    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container"
Jul 28 17:41:42.633635 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.633516    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kserve-container"
Jul 28 17:41:42.633635 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.633526    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" containerName="kube-rbac-proxy"
Jul 28 17:41:42.637153 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.637132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.639942 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.639915    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:41:42.640050 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.639976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:41:42.640136 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.640119    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:41:42.640258 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.640172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 17:41:42.640258 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.640206    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 17:41:42.640258 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.640236    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:41:42.640403 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.640315    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:41:42.650850 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.650830    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"]
Jul 28 17:41:42.807669 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.807635    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-api-access-brdnt\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.807669 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.807686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9182508-ed43-4c4b-89c2-db76d12113c4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.807901 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.807716    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9182508-ed43-4c4b-89c2-db76d12113c4-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.807901 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.807754    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.807901 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.807777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.909084 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.909053    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.909567 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.909127    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-api-access-brdnt\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.909567 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.909163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9182508-ed43-4c4b-89c2-db76d12113c4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.909567 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.909194    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9182508-ed43-4c4b-89c2-db76d12113c4-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.909567 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.909236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.909803 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.909622    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9182508-ed43-4c4b-89c2-db76d12113c4-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.909860 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.909810    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.909945 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.909925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-cabundle-cert\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.911443 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.911420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9182508-ed43-4c4b-89c2-db76d12113c4-proxy-tls\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.918219 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.918200    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-api-access-brdnt\") pod \"isvc-canary-rollback-predictor-b4645bb5c-988l7\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:42.950224 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:42.950197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:43.076814 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:43.076789    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"]
Jul 28 17:41:43.078713 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:41:43.078685    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9182508_ed43_4c4b_89c2_db76d12113c4.slice/crio-bd6d380a64fbbfde95c253ed1eedd1ac61371cf32823f31b0c612801d1717548 WatchSource:0}: Error finding container bd6d380a64fbbfde95c253ed1eedd1ac61371cf32823f31b0c612801d1717548: Status 404 returned error can't find the container with id bd6d380a64fbbfde95c253ed1eedd1ac61371cf32823f31b0c612801d1717548
Jul 28 17:41:43.252453 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:43.252359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" event={"ID":"e9182508-ed43-4c4b-89c2-db76d12113c4","Type":"ContainerStarted","Data":"0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2"}
Jul 28 17:41:43.252453 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:43.252437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" event={"ID":"e9182508-ed43-4c4b-89c2-db76d12113c4","Type":"ContainerStarted","Data":"bd6d380a64fbbfde95c253ed1eedd1ac61371cf32823f31b0c612801d1717548"}
Jul 28 17:41:43.905030 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:43.904994    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="041e9779-cae3-4fac-887d-a6f0adf14247" path="/var/lib/kubelet/pods/041e9779-cae3-4fac-887d-a6f0adf14247/volumes"
Jul 28 17:41:47.270827 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:47.270794    2577 generic.go:356] "Generic (PLEG): container finished" podID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerID="0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2" exitCode=0
Jul 28 17:41:47.271193 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:47.270867    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" event={"ID":"e9182508-ed43-4c4b-89c2-db76d12113c4","Type":"ContainerDied","Data":"0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2"}
Jul 28 17:41:48.276670 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:48.276632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" event={"ID":"e9182508-ed43-4c4b-89c2-db76d12113c4","Type":"ContainerStarted","Data":"e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd"}
Jul 28 17:41:48.277182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:48.276704    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" event={"ID":"e9182508-ed43-4c4b-89c2-db76d12113c4","Type":"ContainerStarted","Data":"1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57"}
Jul 28 17:41:48.277182 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:48.276917    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:48.299151 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:48.299106    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podStartSLOduration=2.378605755 podStartE2EDuration="6.29908982s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.920484065s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:48.297489791 +0000 UTC m=+1866.941459300" watchObservedRunningTime="2026-07-28 17:41:48.29908982 +0000 UTC m=+1866.943059376"
Jul 28 17:41:49.281140 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:49.281109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:49.282474 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:49.282445    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:41:50.284788 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:50.284751    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:41:55.290092 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:55.290060    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:41:55.290728 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:41:55.290694    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:42:05.290926 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:42:05.290843    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:42:15.290876 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:42:15.290834    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:42:25.290845 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:42:25.290803    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:42:35.291630 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:42:35.291587    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:42:45.290978 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:42:45.290938    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:42:55.291067 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:42:55.291024    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:42:59.904752 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:42:59.904725    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:44:22.792021 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:22.791990    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"]
Jul 28 17:44:22.792566 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:22.792264    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container" containerID="cri-o://1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57" gracePeriod=30
Jul 28 17:44:22.792566 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:22.792324    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kube-rbac-proxy" containerID="cri-o://e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd" gracePeriod=30
Jul 28 17:44:23.906480 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:23.906445    2577 generic.go:356] "Generic (PLEG): container finished" podID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerID="e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd" exitCode=2
Jul 28 17:44:23.906855 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:23.906493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" event={"ID":"e9182508-ed43-4c4b-89c2-db76d12113c4","Type":"ContainerDied","Data":"e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd"}
Jul 28 17:44:25.285440 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:25.285396    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 28 17:44:27.134474 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.134451    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:44:27.179898 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.179878    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-cabundle-cert\") pod \"e9182508-ed43-4c4b-89c2-db76d12113c4\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") "
Jul 28 17:44:27.180034 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.179943    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-api-access-brdnt\") pod \"e9182508-ed43-4c4b-89c2-db76d12113c4\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") "
Jul 28 17:44:27.180034 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.179970    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9182508-ed43-4c4b-89c2-db76d12113c4-proxy-tls\") pod \"e9182508-ed43-4c4b-89c2-db76d12113c4\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") "
Jul 28 17:44:27.180034 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.179999    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-rbac-proxy-sar-config\") pod \"e9182508-ed43-4c4b-89c2-db76d12113c4\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") "
Jul 28 17:44:27.180190 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.180072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9182508-ed43-4c4b-89c2-db76d12113c4-kserve-provision-location\") pod \"e9182508-ed43-4c4b-89c2-db76d12113c4\" (UID: \"e9182508-ed43-4c4b-89c2-db76d12113c4\") "
Jul 28 17:44:27.180361 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.180224    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e9182508-ed43-4c4b-89c2-db76d12113c4" (UID: "e9182508-ed43-4c4b-89c2-db76d12113c4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:44:27.180430 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.180399    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9182508-ed43-4c4b-89c2-db76d12113c4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e9182508-ed43-4c4b-89c2-db76d12113c4" (UID: "e9182508-ed43-4c4b-89c2-db76d12113c4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:44:27.180430 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.180409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e9182508-ed43-4c4b-89c2-db76d12113c4" (UID: "e9182508-ed43-4c4b-89c2-db76d12113c4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:44:27.181868 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.181850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-api-access-brdnt" (OuterVolumeSpecName: "kube-api-access-brdnt") pod "e9182508-ed43-4c4b-89c2-db76d12113c4" (UID: "e9182508-ed43-4c4b-89c2-db76d12113c4"). InnerVolumeSpecName "kube-api-access-brdnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:44:27.181954 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.181903    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9182508-ed43-4c4b-89c2-db76d12113c4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e9182508-ed43-4c4b-89c2-db76d12113c4" (UID: "e9182508-ed43-4c4b-89c2-db76d12113c4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:44:27.280634 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.280569    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e9182508-ed43-4c4b-89c2-db76d12113c4-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:44:27.280634 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.280589    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:44:27.280634 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.280600    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brdnt\" (UniqueName: \"kubernetes.io/projected/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-api-access-brdnt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:44:27.280634 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.280609    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e9182508-ed43-4c4b-89c2-db76d12113c4-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:44:27.280634 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.280619    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e9182508-ed43-4c4b-89c2-db76d12113c4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:44:27.922986 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.922955    2577 generic.go:356] "Generic (PLEG): container finished" podID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerID="1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57" exitCode=0
Jul 28 17:44:27.923154 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.923025    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"
Jul 28 17:44:27.923154 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.923029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" event={"ID":"e9182508-ed43-4c4b-89c2-db76d12113c4","Type":"ContainerDied","Data":"1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57"}
Jul 28 17:44:27.923233 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.923192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7" event={"ID":"e9182508-ed43-4c4b-89c2-db76d12113c4","Type":"ContainerDied","Data":"bd6d380a64fbbfde95c253ed1eedd1ac61371cf32823f31b0c612801d1717548"}
Jul 28 17:44:27.923294 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.923260    2577 scope.go:117] "RemoveContainer" containerID="e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd"
Jul 28 17:44:27.935383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.935363    2577 scope.go:117] "RemoveContainer" containerID="1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57"
Jul 28 17:44:27.942697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.942664    2577 scope.go:117] "RemoveContainer" containerID="0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2"
Jul 28 17:44:27.951546 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.951523    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"]
Jul 28 17:44:27.953347 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.953330    2577 scope.go:117] "RemoveContainer" containerID="e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd"
Jul 28 17:44:27.953689 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:44:27.953656    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd\": container with ID starting with e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd not found: ID does not exist" containerID="e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd"
Jul 28 17:44:27.953783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.953696    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd"} err="failed to get container status \"e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd\": rpc error: code = NotFound desc = could not find container \"e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd\": container with ID starting with e56f2472af09b7907aaa2236e79f24568b17f1334dc60021a64a4b5c9bdcd2fd not found: ID does not exist"
Jul 28 17:44:27.953783 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.953713    2577 scope.go:117] "RemoveContainer" containerID="1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57"
Jul 28 17:44:27.954030 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:44:27.954004    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57\": container with ID starting with 1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57 not found: ID does not exist" containerID="1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57"
Jul 28 17:44:27.954093 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.954036    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57"} err="failed to get container status \"1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57\": rpc error: code = NotFound desc = could not find container \"1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57\": container with ID starting with 1e5154b554fc75ecacab6b29896a6301ef0afa14769ca5290293d5065e671c57 not found: ID does not exist"
Jul 28 17:44:27.954093 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.954051    2577 scope.go:117] "RemoveContainer" containerID="0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2"
Jul 28 17:44:27.954453 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:44:27.954430    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2\": container with ID starting with 0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2 not found: ID does not exist" containerID="0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2"
Jul 28 17:44:27.954540 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.954458    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2"} err="failed to get container status \"0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2\": rpc error: code = NotFound desc = could not find container \"0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2\": container with ID starting with 0be2368dbdd22c6a37bcce5ef38f6d35ba1dc0e76bbac85e915c84c5b4c987f2 not found: ID does not exist"
Jul 28 17:44:27.961775 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:27.961754    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-b4645bb5c-988l7"]
Jul 28 17:44:28.007991 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.007967    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"]
Jul 28 17:44:28.008299 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.008286    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container"
Jul 28 17:44:28.008349 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.008301    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container"
Jul 28 17:44:28.008349 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.008315    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kube-rbac-proxy"
Jul 28 17:44:28.008349 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.008320    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kube-rbac-proxy"
Jul 28 17:44:28.008349 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.008338    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="storage-initializer"
Jul 28 17:44:28.008349 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.008343    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="storage-initializer"
Jul 28 17:44:28.008519 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.008389    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kserve-container"
Jul 28 17:44:28.008519 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.008400    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" containerName="kube-rbac-proxy"
Jul 28 17:44:28.011264 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.011237    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.014584 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.014565    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:44:28.014959 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.014942    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:44:28.015077 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.014985    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 17:44:28.015077 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.015008    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:44:28.015176 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.015159    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 17:44:28.015221 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.015203    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:44:28.015289 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.015278    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:44:28.023817 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.023796    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"]
Jul 28 17:44:28.085898 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.085871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50e4db4-991a-4e5d-b441-cd87ac885cac-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.086022 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.085908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b50e4db4-991a-4e5d-b441-cd87ac885cac-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.086022 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.085950    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fnb\" (UniqueName: \"kubernetes.io/projected/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-api-access-x7fnb\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.086119 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.086022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.086119 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.086061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.186743 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.186666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fnb\" (UniqueName: \"kubernetes.io/projected/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-api-access-x7fnb\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.186743 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.186716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.187195 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.186761    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.187195 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.186819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50e4db4-991a-4e5d-b441-cd87ac885cac-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.187195 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.186855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b50e4db4-991a-4e5d-b441-cd87ac885cac-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.187195 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:44:28.186975    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 17:44:28.187195 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:44:28.187050    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50e4db4-991a-4e5d-b441-cd87ac885cac-proxy-tls podName:b50e4db4-991a-4e5d-b441-cd87ac885cac nodeName:}" failed. No retries permitted until 2026-07-28 17:44:28.687028166 +0000 UTC m=+2027.330997640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b50e4db4-991a-4e5d-b441-cd87ac885cac-proxy-tls") pod "isvc-canary-stop-predictor-5bf5667989-mvz6n" (UID: "b50e4db4-991a-4e5d-b441-cd87ac885cac") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 17:44:28.187434 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.187284    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b50e4db4-991a-4e5d-b441-cd87ac885cac-kserve-provision-location\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.187470 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.187435    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.187510 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.187494    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-cabundle-cert\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.199223 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.199204    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fnb\" (UniqueName: \"kubernetes.io/projected/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-api-access-x7fnb\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.690822 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.690793    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50e4db4-991a-4e5d-b441-cd87ac885cac-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.693160 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.693141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50e4db4-991a-4e5d-b441-cd87ac885cac-proxy-tls\") pod \"isvc-canary-stop-predictor-5bf5667989-mvz6n\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:28.924059 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:28.924027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:29.052928 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:29.052906    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"]
Jul 28 17:44:29.055149 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:44:29.055124    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50e4db4_991a_4e5d_b441_cd87ac885cac.slice/crio-0ad21655ce95aa1e31feb2255555b09ceac2e17463fa580d16a3939750124ec2 WatchSource:0}: Error finding container 0ad21655ce95aa1e31feb2255555b09ceac2e17463fa580d16a3939750124ec2: Status 404 returned error can't find the container with id 0ad21655ce95aa1e31feb2255555b09ceac2e17463fa580d16a3939750124ec2
Jul 28 17:44:29.904568 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:29.904538    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9182508-ed43-4c4b-89c2-db76d12113c4" path="/var/lib/kubelet/pods/e9182508-ed43-4c4b-89c2-db76d12113c4/volumes"
Jul 28 17:44:29.934526 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:29.934497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" event={"ID":"b50e4db4-991a-4e5d-b441-cd87ac885cac","Type":"ContainerStarted","Data":"d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8"}
Jul 28 17:44:29.934801 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:29.934542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" event={"ID":"b50e4db4-991a-4e5d-b441-cd87ac885cac","Type":"ContainerStarted","Data":"0ad21655ce95aa1e31feb2255555b09ceac2e17463fa580d16a3939750124ec2"}
Jul 28 17:44:33.952470 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:33.952435    2577 generic.go:356] "Generic (PLEG): container finished" podID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerID="d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8" exitCode=0
Jul 28 17:44:33.952929 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:33.952510    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" event={"ID":"b50e4db4-991a-4e5d-b441-cd87ac885cac","Type":"ContainerDied","Data":"d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8"}
Jul 28 17:44:34.959443 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:34.959407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" event={"ID":"b50e4db4-991a-4e5d-b441-cd87ac885cac","Type":"ContainerStarted","Data":"4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2"}
Jul 28 17:44:34.959924 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:34.959477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" event={"ID":"b50e4db4-991a-4e5d-b441-cd87ac885cac","Type":"ContainerStarted","Data":"a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c"}
Jul 28 17:44:34.959924 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:34.959695    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:34.983611 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:34.983566    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podStartSLOduration=4.123668434 podStartE2EDuration="7.983548121s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.859879687s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:34.981491482 +0000 UTC m=+2033.625460972" watchObservedRunningTime="2026-07-28 17:44:34.983548121 +0000 UTC m=+2033.627517610"
Jul 28 17:44:35.963880 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:35.963851    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:35.965158 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:35.965127    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:44:36.968151 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:36.968115    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:44:41.972755 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:41.972729    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:44:41.973181 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:41.973155    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:44:51.973484 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:44:51.973440    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:45:01.974065 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:01.974020    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:45:11.973910 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:11.973832    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:45:21.973179 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:21.973143    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:45:31.973472 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:31.973432    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:45:41.912985 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:41.912955    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:45:41.913427 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:41.913051    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:45:41.973281 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:41.973225    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:45:51.973383 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:51.973355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:45:58.036185 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:58.036155    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"]
Jul 28 17:45:58.036700 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:58.036420    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" containerID="cri-o://a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c" gracePeriod=30
Jul 28 17:45:58.036700 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:58.036438    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kube-rbac-proxy" containerID="cri-o://4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2" gracePeriod=30
Jul 28 17:45:58.292226 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:58.292148    2577 generic.go:356] "Generic (PLEG): container finished" podID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerID="4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2" exitCode=2
Jul 28 17:45:58.292377 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:45:58.292219    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" event={"ID":"b50e4db4-991a-4e5d-b441-cd87ac885cac","Type":"ContainerDied","Data":"4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2"}
Jul 28 17:46:01.968917 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:01.968877    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 17:46:01.973220 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:01.973196    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:46:02.475774 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.475752    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:46:02.603560 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.603491    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b50e4db4-991a-4e5d-b441-cd87ac885cac-kserve-provision-location\") pod \"b50e4db4-991a-4e5d-b441-cd87ac885cac\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") "
Jul 28 17:46:02.603691 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.603576    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50e4db4-991a-4e5d-b441-cd87ac885cac-proxy-tls\") pod \"b50e4db4-991a-4e5d-b441-cd87ac885cac\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") "
Jul 28 17:46:02.603691 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.603603    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7fnb\" (UniqueName: \"kubernetes.io/projected/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-api-access-x7fnb\") pod \"b50e4db4-991a-4e5d-b441-cd87ac885cac\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") "
Jul 28 17:46:02.603691 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.603657    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-cabundle-cert\") pod \"b50e4db4-991a-4e5d-b441-cd87ac885cac\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") "
Jul 28 17:46:02.603691 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.603682    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-rbac-proxy-sar-config\") pod \"b50e4db4-991a-4e5d-b441-cd87ac885cac\" (UID: \"b50e4db4-991a-4e5d-b441-cd87ac885cac\") "
Jul 28 17:46:02.603893 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.603763    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50e4db4-991a-4e5d-b441-cd87ac885cac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b50e4db4-991a-4e5d-b441-cd87ac885cac" (UID: "b50e4db4-991a-4e5d-b441-cd87ac885cac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:02.603947 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.603915    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b50e4db4-991a-4e5d-b441-cd87ac885cac-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:46:02.604030 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.604010    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b50e4db4-991a-4e5d-b441-cd87ac885cac" (UID: "b50e4db4-991a-4e5d-b441-cd87ac885cac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:02.604110 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.604090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b50e4db4-991a-4e5d-b441-cd87ac885cac" (UID: "b50e4db4-991a-4e5d-b441-cd87ac885cac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:02.605623 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.605602    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-api-access-x7fnb" (OuterVolumeSpecName: "kube-api-access-x7fnb") pod "b50e4db4-991a-4e5d-b441-cd87ac885cac" (UID: "b50e4db4-991a-4e5d-b441-cd87ac885cac"). InnerVolumeSpecName "kube-api-access-x7fnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:02.605718 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.605695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50e4db4-991a-4e5d-b441-cd87ac885cac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b50e4db4-991a-4e5d-b441-cd87ac885cac" (UID: "b50e4db4-991a-4e5d-b441-cd87ac885cac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:02.704864 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.704840    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:46:02.704864 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.704861    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:46:02.704992 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.704870    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b50e4db4-991a-4e5d-b441-cd87ac885cac-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:46:02.704992 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:02.704879    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7fnb\" (UniqueName: \"kubernetes.io/projected/b50e4db4-991a-4e5d-b441-cd87ac885cac-kube-api-access-x7fnb\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:46:03.314285 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.314236    2577 generic.go:356] "Generic (PLEG): container finished" podID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerID="a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c" exitCode=0
Jul 28 17:46:03.314795 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.314303    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"
Jul 28 17:46:03.314795 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.314339    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" event={"ID":"b50e4db4-991a-4e5d-b441-cd87ac885cac","Type":"ContainerDied","Data":"a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c"}
Jul 28 17:46:03.314795 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.314396    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n" event={"ID":"b50e4db4-991a-4e5d-b441-cd87ac885cac","Type":"ContainerDied","Data":"0ad21655ce95aa1e31feb2255555b09ceac2e17463fa580d16a3939750124ec2"}
Jul 28 17:46:03.314795 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.314445    2577 scope.go:117] "RemoveContainer" containerID="4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2"
Jul 28 17:46:03.326734 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.326688    2577 scope.go:117] "RemoveContainer" containerID="a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c"
Jul 28 17:46:03.333599 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.333574    2577 scope.go:117] "RemoveContainer" containerID="d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8"
Jul 28 17:46:03.342802 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.342784    2577 scope.go:117] "RemoveContainer" containerID="4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2"
Jul 28 17:46:03.343077 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:46:03.343049    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2\": container with ID starting with 4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2 not found: ID does not exist" containerID="4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2"
Jul 28 17:46:03.343139 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.343079    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2"} err="failed to get container status \"4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2\": rpc error: code = NotFound desc = could not find container \"4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2\": container with ID starting with 4f134fbbb1f82d4382aa53bd3e17d686508fcd82aa3936f170ea8ede967f15a2 not found: ID does not exist"
Jul 28 17:46:03.343139 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.343096    2577 scope.go:117] "RemoveContainer" containerID="a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c"
Jul 28 17:46:03.343255 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.343221    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"]
Jul 28 17:46:03.343350 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:46:03.343328    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c\": container with ID starting with a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c not found: ID does not exist" containerID="a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c"
Jul 28 17:46:03.343395 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.343350    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c"} err="failed to get container status \"a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c\": rpc error: code = NotFound desc = could not find container \"a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c\": container with ID starting with a1c1fccb47db7952a4cc152e56fc9ef9f59032c3eaaea335c455fb458895b05c not found: ID does not exist"
Jul 28 17:46:03.343395 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.343365    2577 scope.go:117] "RemoveContainer" containerID="d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8"
Jul 28 17:46:03.343604 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:46:03.343583    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8\": container with ID starting with d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8 not found: ID does not exist" containerID="d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8"
Jul 28 17:46:03.343650 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.343613    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8"} err="failed to get container status \"d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8\": rpc error: code = NotFound desc = could not find container \"d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8\": container with ID starting with d8dafbb84ce64c4a8baa528086a7ba2f133eb7d2952481d9aa2b95db74656db8 not found: ID does not exist"
Jul 28 17:46:03.346220 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.346202    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-5bf5667989-mvz6n"]
Jul 28 17:46:03.903962 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:03.903928    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" path="/var/lib/kubelet/pods/b50e4db4-991a-4e5d-b441-cd87ac885cac/volumes"
Jul 28 17:46:08.370447 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.370418    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"]
Jul 28 17:46:08.370939 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.370875    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container"
Jul 28 17:46:08.370939 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.370893    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container"
Jul 28 17:46:08.370939 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.370910    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="storage-initializer"
Jul 28 17:46:08.370939 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.370918    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="storage-initializer"
Jul 28 17:46:08.370939 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.370932    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kube-rbac-proxy"
Jul 28 17:46:08.370939 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.370940    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kube-rbac-proxy"
Jul 28 17:46:08.371234 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.371028    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kserve-container"
Jul 28 17:46:08.371234 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.371041    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50e4db4-991a-4e5d-b441-cd87ac885cac" containerName="kube-rbac-proxy"
Jul 28 17:46:08.375705 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.375685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.378468 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.378444    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:46:08.378566 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.378447    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:46:08.378616 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.378587    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:46:08.379659 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.379643    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:46:08.379750 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.379740    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:46:08.379804 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.379769    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 17:46:08.379804 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.379780    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 28 17:46:08.384884 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.384864    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"]
Jul 28 17:46:08.442175 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.442147    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.442307 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.442193    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838004a9-26f2-444e-b3d0-2ccb5018bf55-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.442367 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.442336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838004a9-26f2-444e-b3d0-2ccb5018bf55-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.442411 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.442372    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22cn\" (UniqueName: \"kubernetes.io/projected/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-api-access-k22cn\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.442459 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.442419    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.542915 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.542890    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838004a9-26f2-444e-b3d0-2ccb5018bf55-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.543015 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.542971    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838004a9-26f2-444e-b3d0-2ccb5018bf55-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.543060 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:46:08.543012    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 28 17:46:08.543060 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.543022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22cn\" (UniqueName: \"kubernetes.io/projected/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-api-access-k22cn\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.543060 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:46:08.543058    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/838004a9-26f2-444e-b3d0-2ccb5018bf55-proxy-tls podName:838004a9-26f2-444e-b3d0-2ccb5018bf55 nodeName:}" failed. No retries permitted until 2026-07-28 17:46:09.043043461 +0000 UTC m=+2127.687012929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/838004a9-26f2-444e-b3d0-2ccb5018bf55-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-74w62" (UID: "838004a9-26f2-444e-b3d0-2ccb5018bf55") : secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 28 17:46:08.543208 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.543071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.543208 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.543109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.543315 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.543286    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838004a9-26f2-444e-b3d0-2ccb5018bf55-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.543664 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.543646    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.543719 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.543702    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:08.554073 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:08.554053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22cn\" (UniqueName: \"kubernetes.io/projected/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-api-access-k22cn\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:09.048454 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:09.048415    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838004a9-26f2-444e-b3d0-2ccb5018bf55-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:09.050803 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:09.050778    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838004a9-26f2-444e-b3d0-2ccb5018bf55-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-74w62\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:09.287172 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:09.287137    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:09.414629 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:09.414589    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"]
Jul 28 17:46:09.416814 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:46:09.416789    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838004a9_26f2_444e_b3d0_2ccb5018bf55.slice/crio-e184ccd1356efdaeb284b19833803636d2f8a603e3c70d0d90b02f8eff99fa9c WatchSource:0}: Error finding container e184ccd1356efdaeb284b19833803636d2f8a603e3c70d0d90b02f8eff99fa9c: Status 404 returned error can't find the container with id e184ccd1356efdaeb284b19833803636d2f8a603e3c70d0d90b02f8eff99fa9c
Jul 28 17:46:10.346546 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:10.346512    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" event={"ID":"838004a9-26f2-444e-b3d0-2ccb5018bf55","Type":"ContainerStarted","Data":"3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a"}
Jul 28 17:46:10.346714 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:10.346571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" event={"ID":"838004a9-26f2-444e-b3d0-2ccb5018bf55","Type":"ContainerStarted","Data":"e184ccd1356efdaeb284b19833803636d2f8a603e3c70d0d90b02f8eff99fa9c"}
Jul 28 17:46:13.359260 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:13.359223    2577 generic.go:356] "Generic (PLEG): container finished" podID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerID="3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a" exitCode=0
Jul 28 17:46:13.359597 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:13.359297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" event={"ID":"838004a9-26f2-444e-b3d0-2ccb5018bf55","Type":"ContainerDied","Data":"3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a"}
Jul 28 17:46:13.360790 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:13.360772    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:46:20.396686 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:20.396646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" event={"ID":"838004a9-26f2-444e-b3d0-2ccb5018bf55","Type":"ContainerStarted","Data":"a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5"}
Jul 28 17:46:20.397159 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:20.396708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" event={"ID":"838004a9-26f2-444e-b3d0-2ccb5018bf55","Type":"ContainerStarted","Data":"9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6"}
Jul 28 17:46:20.397159 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:20.396889    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:20.416805 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:20.416759    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podStartSLOduration=2.482360999 podStartE2EDuration="12.41674282s" totalImagesPullingTime="6.110734278s" totalInitContainerRuntime="3.823647543s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:20.416113779 +0000 UTC m=+2139.060083282" watchObservedRunningTime="2026-07-28 17:46:20.41674282 +0000 UTC m=+2139.060712313"
Jul 28 17:46:21.400747 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:21.400716    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:21.401726 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:21.401700    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:46:22.404232 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:22.404195    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:46:27.408338 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:27.408312    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:46:27.408854 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:27.408806    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:46:37.409740 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:37.409650    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:46:47.409531 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:47.409488    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:46:57.408963 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:46:57.408920    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:47:07.409283 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:07.409222    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:47:17.408983 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:17.408935    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:47:27.409446 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:27.409405    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 28 17:47:37.409355 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:37.409323    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:47:38.847633 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:38.847599    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"]
Jul 28 17:47:38.848086 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:38.847875    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container" containerID="cri-o://9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6" gracePeriod=30
Jul 28 17:47:38.848086 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:38.847943    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kube-rbac-proxy" containerID="cri-o://a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5" gracePeriod=30
Jul 28 17:47:39.716817 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:39.716777    2577 generic.go:356] "Generic (PLEG): container finished" podID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerID="a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5" exitCode=2
Jul 28 17:47:39.716999 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:39.716830    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" event={"ID":"838004a9-26f2-444e-b3d0-2ccb5018bf55","Type":"ContainerDied","Data":"a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5"}
Jul 28 17:47:42.405518 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:42.405472    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 28 17:47:43.692798 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.692773    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:47:43.733484 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.733451    2577 generic.go:356] "Generic (PLEG): container finished" podID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerID="9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6" exitCode=0
Jul 28 17:47:43.733636 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.733508    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"
Jul 28 17:47:43.733636 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.733525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" event={"ID":"838004a9-26f2-444e-b3d0-2ccb5018bf55","Type":"ContainerDied","Data":"9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6"}
Jul 28 17:47:43.733636 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.733585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62" event={"ID":"838004a9-26f2-444e-b3d0-2ccb5018bf55","Type":"ContainerDied","Data":"e184ccd1356efdaeb284b19833803636d2f8a603e3c70d0d90b02f8eff99fa9c"}
Jul 28 17:47:43.733636 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.733613    2577 scope.go:117] "RemoveContainer" containerID="a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5"
Jul 28 17:47:43.741742 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.741723    2577 scope.go:117] "RemoveContainer" containerID="9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6"
Jul 28 17:47:43.748995 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.748979    2577 scope.go:117] "RemoveContainer" containerID="3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a"
Jul 28 17:47:43.756527 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.756510    2577 scope.go:117] "RemoveContainer" containerID="a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5"
Jul 28 17:47:43.756773 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:47:43.756751    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5\": container with ID starting with a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5 not found: ID does not exist" containerID="a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5"
Jul 28 17:47:43.756818 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.756785    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5"} err="failed to get container status \"a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5\": rpc error: code = NotFound desc = could not find container \"a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5\": container with ID starting with a6f7db3dd4e379b278ddee8a9b4064112e67cbbd721525237fbfd332b435fca5 not found: ID does not exist"
Jul 28 17:47:43.756818 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.756810    2577 scope.go:117] "RemoveContainer" containerID="9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6"
Jul 28 17:47:43.757031 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:47:43.757015    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6\": container with ID starting with 9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6 not found: ID does not exist" containerID="9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6"
Jul 28 17:47:43.757083 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.757038    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6"} err="failed to get container status \"9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6\": rpc error: code = NotFound desc = could not find container \"9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6\": container with ID starting with 9873da5181f0d45588be3c70a2646e336e04b5d9232ece70a47478b6f4a127a6 not found: ID does not exist"
Jul 28 17:47:43.757083 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.757054    2577 scope.go:117] "RemoveContainer" containerID="3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a"
Jul 28 17:47:43.757295 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:47:43.757275    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a\": container with ID starting with 3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a not found: ID does not exist" containerID="3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a"
Jul 28 17:47:43.757348 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.757304    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a"} err="failed to get container status \"3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a\": rpc error: code = NotFound desc = could not find container \"3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a\": container with ID starting with 3d2e6fbc6a91dddb0bd2313e34c291d4d078299057ec723129146aa5edc6e66a not found: ID does not exist"
Jul 28 17:47:43.815327 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.815214    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k22cn\" (UniqueName: \"kubernetes.io/projected/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-api-access-k22cn\") pod \"838004a9-26f2-444e-b3d0-2ccb5018bf55\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") "
Jul 28 17:47:43.815327 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.815304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838004a9-26f2-444e-b3d0-2ccb5018bf55-proxy-tls\") pod \"838004a9-26f2-444e-b3d0-2ccb5018bf55\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") "
Jul 28 17:47:43.815539 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.815461    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-rbac-proxy-sar-config\") pod \"838004a9-26f2-444e-b3d0-2ccb5018bf55\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") "
Jul 28 17:47:43.815539 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.815505    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838004a9-26f2-444e-b3d0-2ccb5018bf55-kserve-provision-location\") pod \"838004a9-26f2-444e-b3d0-2ccb5018bf55\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") "
Jul 28 17:47:43.815641 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.815552    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-cabundle-cert\") pod \"838004a9-26f2-444e-b3d0-2ccb5018bf55\" (UID: \"838004a9-26f2-444e-b3d0-2ccb5018bf55\") "
Jul 28 17:47:43.815869 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.815826    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/838004a9-26f2-444e-b3d0-2ccb5018bf55-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "838004a9-26f2-444e-b3d0-2ccb5018bf55" (UID: "838004a9-26f2-444e-b3d0-2ccb5018bf55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:47:43.815869 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.815826    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "838004a9-26f2-444e-b3d0-2ccb5018bf55" (UID: "838004a9-26f2-444e-b3d0-2ccb5018bf55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:43.816009 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.815881    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "838004a9-26f2-444e-b3d0-2ccb5018bf55" (UID: "838004a9-26f2-444e-b3d0-2ccb5018bf55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:43.817421 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.817399    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/838004a9-26f2-444e-b3d0-2ccb5018bf55-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "838004a9-26f2-444e-b3d0-2ccb5018bf55" (UID: "838004a9-26f2-444e-b3d0-2ccb5018bf55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:43.817507 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.817449    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-api-access-k22cn" (OuterVolumeSpecName: "kube-api-access-k22cn") pod "838004a9-26f2-444e-b3d0-2ccb5018bf55" (UID: "838004a9-26f2-444e-b3d0-2ccb5018bf55"). InnerVolumeSpecName "kube-api-access-k22cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:43.916498 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.916471    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/838004a9-26f2-444e-b3d0-2ccb5018bf55-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:47:43.916498 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.916501    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:47:43.916653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.916516    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/838004a9-26f2-444e-b3d0-2ccb5018bf55-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:47:43.916653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.916531    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/838004a9-26f2-444e-b3d0-2ccb5018bf55-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:47:43.916653 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:43.916546    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k22cn\" (UniqueName: \"kubernetes.io/projected/838004a9-26f2-444e-b3d0-2ccb5018bf55-kube-api-access-k22cn\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:47:44.057707 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:44.057672    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"]
Jul 28 17:47:44.061054 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:44.061024    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-74w62"]
Jul 28 17:47:45.904602 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:47:45.904565    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" path="/var/lib/kubelet/pods/838004a9-26f2-444e-b3d0-2ccb5018bf55/volumes"
Jul 28 17:49:09.709006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.708968    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"]
Jul 28 17:49:09.709494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.709297    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="storage-initializer"
Jul 28 17:49:09.709494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.709310    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="storage-initializer"
Jul 28 17:49:09.709494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.709328    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container"
Jul 28 17:49:09.709494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.709334    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container"
Jul 28 17:49:09.709494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.709341    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kube-rbac-proxy"
Jul 28 17:49:09.709494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.709347    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kube-rbac-proxy"
Jul 28 17:49:09.709494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.709399    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kube-rbac-proxy"
Jul 28 17:49:09.709494 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.709406    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="838004a9-26f2-444e-b3d0-2ccb5018bf55" containerName="kserve-container"
Jul 28 17:49:09.712480 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.712464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.715277 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.715221    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:49:09.715455 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.715410    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:49:09.715577 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.715472    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:49:09.715577 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.715524    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:49:09.716451 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.716431    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:49:09.716549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.716453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:49:09.716549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.716471    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jul 28 17:49:09.724054 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.724032    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"]
Jul 28 17:49:09.798500 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.798458    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a289dda4-329e-4220-a675-9c6dbf0aebc9-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.798500 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.798506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a289dda4-329e-4220-a675-9c6dbf0aebc9-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.798752 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.798570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.798752 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.798610    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.798752 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.798646    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjw29\" (UniqueName: \"kubernetes.io/projected/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-api-access-xjw29\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.899854 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.899814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.899854 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.899859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.900080 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.899891    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjw29\" (UniqueName: \"kubernetes.io/projected/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-api-access-xjw29\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.900080 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.899931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a289dda4-329e-4220-a675-9c6dbf0aebc9-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.900080 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.899959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a289dda4-329e-4220-a675-9c6dbf0aebc9-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.900578 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.900553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a289dda4-329e-4220-a675-9c6dbf0aebc9-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.900676 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.900618    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.900718 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.900691    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.903019 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.902997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a289dda4-329e-4220-a675-9c6dbf0aebc9-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:09.911828 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:09.911807    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjw29\" (UniqueName: \"kubernetes.io/projected/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-api-access-xjw29\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:10.022534 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:10.022472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:49:10.151943 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:10.151915    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"]
Jul 28 17:49:10.156767 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:49:10.156733    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda289dda4_329e_4220_a675_9c6dbf0aebc9.slice/crio-ad509458ddac3ffca583ce8053c16d74c70ea16566c23929492669c5e674a783 WatchSource:0}: Error finding container ad509458ddac3ffca583ce8053c16d74c70ea16566c23929492669c5e674a783: Status 404 returned error can't find the container with id ad509458ddac3ffca583ce8053c16d74c70ea16566c23929492669c5e674a783
Jul 28 17:49:11.083990 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:11.083952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" event={"ID":"a289dda4-329e-4220-a675-9c6dbf0aebc9","Type":"ContainerStarted","Data":"a4169cd81cd9370ff4d63a0dc728ab69539de0431305991debe5f092101edcc5"}
Jul 28 17:49:11.084407 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:11.084047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" event={"ID":"a289dda4-329e-4220-a675-9c6dbf0aebc9","Type":"ContainerStarted","Data":"ad509458ddac3ffca583ce8053c16d74c70ea16566c23929492669c5e674a783"}
Jul 28 17:49:14.096880 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:14.096848    2577 generic.go:356] "Generic (PLEG): container finished" podID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerID="a4169cd81cd9370ff4d63a0dc728ab69539de0431305991debe5f092101edcc5" exitCode=0
Jul 28 17:49:14.097300 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:49:14.096893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" event={"ID":"a289dda4-329e-4220-a675-9c6dbf0aebc9","Type":"ContainerDied","Data":"a4169cd81cd9370ff4d63a0dc728ab69539de0431305991debe5f092101edcc5"}
Jul 28 17:51:15.917625 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:15.917595    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:51:15.918820 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:15.918798    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:51:17.701291 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:17.701234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" event={"ID":"a289dda4-329e-4220-a675-9c6dbf0aebc9","Type":"ContainerStarted","Data":"3fc6962fe4868885fc3d32dd7d96f63be0b40ba00e7998606aa49987c2f76977"}
Jul 28 17:51:17.701705 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:17.701328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" event={"ID":"a289dda4-329e-4220-a675-9c6dbf0aebc9","Type":"ContainerStarted","Data":"b469a1e20528f6b861975e6ca6305243bb3c87d0376ee3dd5ead06e5fb7ffc23"}
Jul 28 17:51:17.701705 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:17.701356    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:51:17.701705 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:17.701375    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:51:17.735316 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:17.735262    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" podStartSLOduration=2.540718821 podStartE2EDuration="2m8.735225681s" totalImagesPullingTime="2m2.498586748s" totalInitContainerRuntime="3.695920112s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:51:17.733827406 +0000 UTC m=+2436.377796889" watchObservedRunningTime="2026-07-28 17:51:17.735225681 +0000 UTC m=+2436.379195230"
Jul 28 17:51:23.711057 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:23.711028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:51:53.715568 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:53.715532    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:51:59.925589 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:59.925554    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"]
Jul 28 17:51:59.926000 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:59.925827    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="kserve-container" containerID="cri-o://3fc6962fe4868885fc3d32dd7d96f63be0b40ba00e7998606aa49987c2f76977" gracePeriod=30
Jul 28 17:51:59.926000 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:51:59.925875    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="kube-rbac-proxy" containerID="cri-o://b469a1e20528f6b861975e6ca6305243bb3c87d0376ee3dd5ead06e5fb7ffc23" gracePeriod=30
Jul 28 17:52:00.887505 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:00.887462    2577 generic.go:356] "Generic (PLEG): container finished" podID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerID="b469a1e20528f6b861975e6ca6305243bb3c87d0376ee3dd5ead06e5fb7ffc23" exitCode=2
Jul 28 17:52:00.887505 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:00.887492    2577 generic.go:356] "Generic (PLEG): container finished" podID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerID="3fc6962fe4868885fc3d32dd7d96f63be0b40ba00e7998606aa49987c2f76977" exitCode=0
Jul 28 17:52:00.887697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:00.887533    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" event={"ID":"a289dda4-329e-4220-a675-9c6dbf0aebc9","Type":"ContainerDied","Data":"b469a1e20528f6b861975e6ca6305243bb3c87d0376ee3dd5ead06e5fb7ffc23"}
Jul 28 17:52:00.887697 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:00.887608    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" event={"ID":"a289dda4-329e-4220-a675-9c6dbf0aebc9","Type":"ContainerDied","Data":"3fc6962fe4868885fc3d32dd7d96f63be0b40ba00e7998606aa49987c2f76977"}
Jul 28 17:52:00.974447 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:00.974423    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:52:01.092348 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.092227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-cabundle-cert\") pod \"a289dda4-329e-4220-a675-9c6dbf0aebc9\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") "
Jul 28 17:52:01.092348 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.092324    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a289dda4-329e-4220-a675-9c6dbf0aebc9-kserve-provision-location\") pod \"a289dda4-329e-4220-a675-9c6dbf0aebc9\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") "
Jul 28 17:52:01.092348 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.092349    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a289dda4-329e-4220-a675-9c6dbf0aebc9-proxy-tls\") pod \"a289dda4-329e-4220-a675-9c6dbf0aebc9\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") "
Jul 28 17:52:01.092633 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.092365    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjw29\" (UniqueName: \"kubernetes.io/projected/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-api-access-xjw29\") pod \"a289dda4-329e-4220-a675-9c6dbf0aebc9\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") "
Jul 28 17:52:01.092633 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.092399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-rbac-proxy-sar-config\") pod \"a289dda4-329e-4220-a675-9c6dbf0aebc9\" (UID: \"a289dda4-329e-4220-a675-9c6dbf0aebc9\") "
Jul 28 17:52:01.092868 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.092740    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a289dda4-329e-4220-a675-9c6dbf0aebc9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a289dda4-329e-4220-a675-9c6dbf0aebc9" (UID: "a289dda4-329e-4220-a675-9c6dbf0aebc9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:52:01.092868 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.092668    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a289dda4-329e-4220-a675-9c6dbf0aebc9" (UID: "a289dda4-329e-4220-a675-9c6dbf0aebc9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:52:01.092868 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.092817    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a289dda4-329e-4220-a675-9c6dbf0aebc9" (UID: "a289dda4-329e-4220-a675-9c6dbf0aebc9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:52:01.094527 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.094509    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a289dda4-329e-4220-a675-9c6dbf0aebc9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a289dda4-329e-4220-a675-9c6dbf0aebc9" (UID: "a289dda4-329e-4220-a675-9c6dbf0aebc9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:52:01.094632 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.094612    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-api-access-xjw29" (OuterVolumeSpecName: "kube-api-access-xjw29") pod "a289dda4-329e-4220-a675-9c6dbf0aebc9" (UID: "a289dda4-329e-4220-a675-9c6dbf0aebc9"). InnerVolumeSpecName "kube-api-access-xjw29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:01.193819 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.193776    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:52:01.193819 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.193809    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a289dda4-329e-4220-a675-9c6dbf0aebc9-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:52:01.193819 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.193818    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a289dda4-329e-4220-a675-9c6dbf0aebc9-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:52:01.193819 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.193828    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjw29\" (UniqueName: \"kubernetes.io/projected/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-api-access-xjw29\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:52:01.194085 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.193837    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a289dda4-329e-4220-a675-9c6dbf0aebc9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:52:01.894145 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.894120    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"
Jul 28 17:52:01.894315 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.894120    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249" event={"ID":"a289dda4-329e-4220-a675-9c6dbf0aebc9","Type":"ContainerDied","Data":"ad509458ddac3ffca583ce8053c16d74c70ea16566c23929492669c5e674a783"}
Jul 28 17:52:01.894315 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.894266    2577 scope.go:117] "RemoveContainer" containerID="b469a1e20528f6b861975e6ca6305243bb3c87d0376ee3dd5ead06e5fb7ffc23"
Jul 28 17:52:01.902711 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.902671    2577 scope.go:117] "RemoveContainer" containerID="3fc6962fe4868885fc3d32dd7d96f63be0b40ba00e7998606aa49987c2f76977"
Jul 28 17:52:01.910607 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.910590    2577 scope.go:117] "RemoveContainer" containerID="a4169cd81cd9370ff4d63a0dc728ab69539de0431305991debe5f092101edcc5"
Jul 28 17:52:01.919183 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.919162    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"]
Jul 28 17:52:01.924606 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:01.924579    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-rc249"]
Jul 28 17:52:03.904158 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:03.904120    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" path="/var/lib/kubelet/pods/a289dda4-329e-4220-a675-9c6dbf0aebc9/volumes"
Jul 28 17:52:30.320021 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.319983    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"]
Jul 28 17:52:30.320516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.320307    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="kserve-container"
Jul 28 17:52:30.320516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.320318    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="kserve-container"
Jul 28 17:52:30.320516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.320332    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="kube-rbac-proxy"
Jul 28 17:52:30.320516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.320337    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="kube-rbac-proxy"
Jul 28 17:52:30.320516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.320344    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="storage-initializer"
Jul 28 17:52:30.320516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.320352    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="storage-initializer"
Jul 28 17:52:30.320516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.320408    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="kube-rbac-proxy"
Jul 28 17:52:30.320516 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.320415    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a289dda4-329e-4220-a675-9c6dbf0aebc9" containerName="kserve-container"
Jul 28 17:52:30.322497 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.322480    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.326001 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.325976    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jul 28 17:52:30.326170 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.326003    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:52:30.326170 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.326005    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:52:30.326338 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.326323    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:52:30.326528 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.326514    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:52:30.327382 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.327364    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 17:52:30.327455 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.327381    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:52:30.330952 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.330929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.331045 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.330976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a842cebf-b8ee-4702-a20e-5b006ebb7261-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.331045 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.331007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a842cebf-b8ee-4702-a20e-5b006ebb7261-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.331124 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.331090    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpb7t\" (UniqueName: \"kubernetes.io/projected/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-api-access-gpb7t\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.331163 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.331133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.337416 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.337395    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"]
Jul 28 17:52:30.431580 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.431538    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpb7t\" (UniqueName: \"kubernetes.io/projected/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-api-access-gpb7t\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.431761 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.431589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.431761 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.431644    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.431761 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.431682    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a842cebf-b8ee-4702-a20e-5b006ebb7261-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.431761 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.431712    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a842cebf-b8ee-4702-a20e-5b006ebb7261-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.431946 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:52:30.431843    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 17:52:30.431946 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:52:30.431909    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a842cebf-b8ee-4702-a20e-5b006ebb7261-proxy-tls podName:a842cebf-b8ee-4702-a20e-5b006ebb7261 nodeName:}" failed. No retries permitted until 2026-07-28 17:52:30.931889126 +0000 UTC m=+2509.575858596 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a842cebf-b8ee-4702-a20e-5b006ebb7261-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" (UID: "a842cebf-b8ee-4702-a20e-5b006ebb7261") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jul 28 17:52:30.432171 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.432145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a842cebf-b8ee-4702-a20e-5b006ebb7261-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.432418 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.432401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.432465 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.432436    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.442004 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.441976    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpb7t\" (UniqueName: \"kubernetes.io/projected/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-api-access-gpb7t\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.936409 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.936373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a842cebf-b8ee-4702-a20e-5b006ebb7261-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:30.938859 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:30.938837    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a842cebf-b8ee-4702-a20e-5b006ebb7261-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:31.232662 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:31.232569    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:31.367523 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:31.367499    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"]
Jul 28 17:52:31.369614 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:52:31.369588    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda842cebf_b8ee_4702_a20e_5b006ebb7261.slice/crio-a51c8567148f2912b518675fa7cdcd6b20244be279b0179ba6a91a90ee2a2d0d WatchSource:0}: Error finding container a51c8567148f2912b518675fa7cdcd6b20244be279b0179ba6a91a90ee2a2d0d: Status 404 returned error can't find the container with id a51c8567148f2912b518675fa7cdcd6b20244be279b0179ba6a91a90ee2a2d0d
Jul 28 17:52:32.023205 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:32.023169    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" event={"ID":"a842cebf-b8ee-4702-a20e-5b006ebb7261","Type":"ContainerStarted","Data":"03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d"}
Jul 28 17:52:32.023392 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:32.023232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" event={"ID":"a842cebf-b8ee-4702-a20e-5b006ebb7261","Type":"ContainerStarted","Data":"a51c8567148f2912b518675fa7cdcd6b20244be279b0179ba6a91a90ee2a2d0d"}
Jul 28 17:52:36.041473 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:36.041437    2577 generic.go:356] "Generic (PLEG): container finished" podID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerID="03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d" exitCode=0
Jul 28 17:52:36.041923 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:36.041499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" event={"ID":"a842cebf-b8ee-4702-a20e-5b006ebb7261","Type":"ContainerDied","Data":"03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d"}
Jul 28 17:52:37.048284 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:37.048234    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" event={"ID":"a842cebf-b8ee-4702-a20e-5b006ebb7261","Type":"ContainerStarted","Data":"8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc"}
Jul 28 17:52:37.048718 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:37.048312    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" event={"ID":"a842cebf-b8ee-4702-a20e-5b006ebb7261","Type":"ContainerStarted","Data":"3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda"}
Jul 28 17:52:37.048718 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:37.048547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:37.073009 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:37.072960    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" podStartSLOduration=3.064273048 podStartE2EDuration="7.072942032s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.008668984s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:37.071158301 +0000 UTC m=+2515.715127792" watchObservedRunningTime="2026-07-28 17:52:37.072942032 +0000 UTC m=+2515.716911521"
Jul 28 17:52:38.053091 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:38.053051    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:52:44.063906 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:52:44.063871    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:53:14.068062 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:14.068028    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:53:20.506127 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:20.506094    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"]
Jul 28 17:53:20.506634 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:20.506356    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="kserve-container" containerID="cri-o://3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda" gracePeriod=30
Jul 28 17:53:20.506634 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:20.506379    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="kube-rbac-proxy" containerID="cri-o://8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc" gracePeriod=30
Jul 28 17:53:21.227550 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.227518    2577 generic.go:356] "Generic (PLEG): container finished" podID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerID="8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc" exitCode=2
Jul 28 17:53:21.227752 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.227599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" event={"ID":"a842cebf-b8ee-4702-a20e-5b006ebb7261","Type":"ContainerDied","Data":"8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc"}
Jul 28 17:53:21.742380 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.742349    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:53:21.772891 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.772865    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-cabundle-cert\") pod \"a842cebf-b8ee-4702-a20e-5b006ebb7261\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") "
Jul 28 17:53:21.773003 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.772911    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a842cebf-b8ee-4702-a20e-5b006ebb7261-proxy-tls\") pod \"a842cebf-b8ee-4702-a20e-5b006ebb7261\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") "
Jul 28 17:53:21.773003 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.772935    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a842cebf-b8ee-4702-a20e-5b006ebb7261-kserve-provision-location\") pod \"a842cebf-b8ee-4702-a20e-5b006ebb7261\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") "
Jul 28 17:53:21.773003 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.772964    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpb7t\" (UniqueName: \"kubernetes.io/projected/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-api-access-gpb7t\") pod \"a842cebf-b8ee-4702-a20e-5b006ebb7261\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") "
Jul 28 17:53:21.773003 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.772995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-rbac-proxy-sar-config\") pod \"a842cebf-b8ee-4702-a20e-5b006ebb7261\" (UID: \"a842cebf-b8ee-4702-a20e-5b006ebb7261\") "
Jul 28 17:53:21.773407 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.773372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a842cebf-b8ee-4702-a20e-5b006ebb7261-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a842cebf-b8ee-4702-a20e-5b006ebb7261" (UID: "a842cebf-b8ee-4702-a20e-5b006ebb7261"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:21.773500 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.773382    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a842cebf-b8ee-4702-a20e-5b006ebb7261" (UID: "a842cebf-b8ee-4702-a20e-5b006ebb7261"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:53:21.773500 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.773404    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a842cebf-b8ee-4702-a20e-5b006ebb7261" (UID: "a842cebf-b8ee-4702-a20e-5b006ebb7261"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:53:21.774920 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.774894    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a842cebf-b8ee-4702-a20e-5b006ebb7261-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a842cebf-b8ee-4702-a20e-5b006ebb7261" (UID: "a842cebf-b8ee-4702-a20e-5b006ebb7261"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:21.775013 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.774995    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-api-access-gpb7t" (OuterVolumeSpecName: "kube-api-access-gpb7t") pod "a842cebf-b8ee-4702-a20e-5b006ebb7261" (UID: "a842cebf-b8ee-4702-a20e-5b006ebb7261"). InnerVolumeSpecName "kube-api-access-gpb7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:21.873652 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.873566    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a842cebf-b8ee-4702-a20e-5b006ebb7261-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:53:21.873652 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.873599    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a842cebf-b8ee-4702-a20e-5b006ebb7261-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:53:21.873652 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.873615    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpb7t\" (UniqueName: \"kubernetes.io/projected/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-api-access-gpb7t\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:53:21.873652 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.873627    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:53:21.873652 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:21.873640    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a842cebf-b8ee-4702-a20e-5b006ebb7261-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:53:22.234293 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.234232    2577 generic.go:356] "Generic (PLEG): container finished" podID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerID="3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda" exitCode=0
Jul 28 17:53:22.234293 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.234281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" event={"ID":"a842cebf-b8ee-4702-a20e-5b006ebb7261","Type":"ContainerDied","Data":"3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda"}
Jul 28 17:53:22.234510 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.234306    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"
Jul 28 17:53:22.234510 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.234345    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb" event={"ID":"a842cebf-b8ee-4702-a20e-5b006ebb7261","Type":"ContainerDied","Data":"a51c8567148f2912b518675fa7cdcd6b20244be279b0179ba6a91a90ee2a2d0d"}
Jul 28 17:53:22.234510 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.234375    2577 scope.go:117] "RemoveContainer" containerID="8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc"
Jul 28 17:53:22.242592 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.242574    2577 scope.go:117] "RemoveContainer" containerID="3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda"
Jul 28 17:53:22.250044 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.250028    2577 scope.go:117] "RemoveContainer" containerID="03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d"
Jul 28 17:53:22.257176 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.257158    2577 scope.go:117] "RemoveContainer" containerID="8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc"
Jul 28 17:53:22.257470 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:53:22.257451    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc\": container with ID starting with 8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc not found: ID does not exist" containerID="8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc"
Jul 28 17:53:22.257558 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.257481    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc"} err="failed to get container status \"8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc\": rpc error: code = NotFound desc = could not find container \"8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc\": container with ID starting with 8275c64d06b4b0211a9bf6493fa04007107d47c0540e6ab8e39f1d4ee22191dc not found: ID does not exist"
Jul 28 17:53:22.257558 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.257505    2577 scope.go:117] "RemoveContainer" containerID="3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda"
Jul 28 17:53:22.257740 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:53:22.257722    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda\": container with ID starting with 3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda not found: ID does not exist" containerID="3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda"
Jul 28 17:53:22.257775 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.257746    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda"} err="failed to get container status \"3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda\": rpc error: code = NotFound desc = could not find container \"3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda\": container with ID starting with 3e56d7af415989692ca54f7176fb8c746bf0278b42d9d0fd196b38b029e67eda not found: ID does not exist"
Jul 28 17:53:22.257775 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.257771    2577 scope.go:117] "RemoveContainer" containerID="03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d"
Jul 28 17:53:22.257989 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:53:22.257971    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d\": container with ID starting with 03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d not found: ID does not exist" containerID="03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d"
Jul 28 17:53:22.258045 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.257998    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d"} err="failed to get container status \"03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d\": rpc error: code = NotFound desc = could not find container \"03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d\": container with ID starting with 03a88403bcc139319c261c731bcc65fa8030d80d1875a196f1dca69ecd333f0d not found: ID does not exist"
Jul 28 17:53:22.266814 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.266793    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"]
Jul 28 17:53:22.277650 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:22.277627    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-tjmfb"]
Jul 28 17:53:23.904689 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:53:23.904658    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" path="/var/lib/kubelet/pods/a842cebf-b8ee-4702-a20e-5b006ebb7261/volumes"
Jul 28 17:54:51.032228 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.032197    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"]
Jul 28 17:54:51.032738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.032540    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="kube-rbac-proxy"
Jul 28 17:54:51.032738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.032552    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="kube-rbac-proxy"
Jul 28 17:54:51.032738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.032562    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="storage-initializer"
Jul 28 17:54:51.032738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.032568    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="storage-initializer"
Jul 28 17:54:51.032738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.032576    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="kserve-container"
Jul 28 17:54:51.032738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.032581    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="kserve-container"
Jul 28 17:54:51.032738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.032662    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="kube-rbac-proxy"
Jul 28 17:54:51.032738 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.032676    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a842cebf-b8ee-4702-a20e-5b006ebb7261" containerName="kserve-container"
Jul 28 17:54:51.039661 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.039622    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.043153 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.043127    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 28 17:54:51.043355 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.043333    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:54:51.043432 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.043407    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 17:54:51.044511 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.044492    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:54:51.044625 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.044563    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:54:51.044625 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.044590    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:54:51.044730 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.044563    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 28 17:54:51.054753 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.054732    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"]
Jul 28 17:54:51.213588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.213551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9850a394-c323-4719-8fd2-a7bae112e4cd-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.213588 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.213598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnrz2\" (UniqueName: \"kubernetes.io/projected/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-api-access-bnrz2\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.213864 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.213671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9850a394-c323-4719-8fd2-a7bae112e4cd-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.213864 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.213712    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.213864 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.213771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.314965 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.314870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.314965 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.314939    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9850a394-c323-4719-8fd2-a7bae112e4cd-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.314965 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.314959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnrz2\" (UniqueName: \"kubernetes.io/projected/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-api-access-bnrz2\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.315290 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.314987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9850a394-c323-4719-8fd2-a7bae112e4cd-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.315290 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.315007    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.315486 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.315461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9850a394-c323-4719-8fd2-a7bae112e4cd-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.315705 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.315682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.315769 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.315692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.317419 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.317403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9850a394-c323-4719-8fd2-a7bae112e4cd-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.326272 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.326232    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnrz2\" (UniqueName: \"kubernetes.io/projected/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-api-access-bnrz2\") pod \"isvc-paddle-predictor-6d5c9fd4f6-lqdfl\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.350290 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.350261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:54:51.485724 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.485702    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"]
Jul 28 17:54:51.488291 ip-10-0-139-190 kubenswrapper[2577]: W0728 17:54:51.488264    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9850a394_c323_4719_8fd2_a7bae112e4cd.slice/crio-d9e505f6d978493c5d11b063171621d0e5aa8721a87ea2a89456bd57ddfd576c WatchSource:0}: Error finding container d9e505f6d978493c5d11b063171621d0e5aa8721a87ea2a89456bd57ddfd576c: Status 404 returned error can't find the container with id d9e505f6d978493c5d11b063171621d0e5aa8721a87ea2a89456bd57ddfd576c
Jul 28 17:54:51.588799 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.588710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" event={"ID":"9850a394-c323-4719-8fd2-a7bae112e4cd","Type":"ContainerStarted","Data":"d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428"}
Jul 28 17:54:51.588799 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:51.588776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" event={"ID":"9850a394-c323-4719-8fd2-a7bae112e4cd","Type":"ContainerStarted","Data":"d9e505f6d978493c5d11b063171621d0e5aa8721a87ea2a89456bd57ddfd576c"}
Jul 28 17:54:56.611479 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:56.611386    2577 generic.go:356] "Generic (PLEG): container finished" podID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerID="d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428" exitCode=0
Jul 28 17:54:56.611479 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:56.611463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" event={"ID":"9850a394-c323-4719-8fd2-a7bae112e4cd","Type":"ContainerDied","Data":"d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428"}
Jul 28 17:54:56.612804 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:54:56.612790    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:55:08.672658 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:08.672622    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" event={"ID":"9850a394-c323-4719-8fd2-a7bae112e4cd","Type":"ContainerStarted","Data":"1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb"}
Jul 28 17:55:08.673084 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:08.672692    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" event={"ID":"9850a394-c323-4719-8fd2-a7bae112e4cd","Type":"ContainerStarted","Data":"9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127"}
Jul 28 17:55:08.673084 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:08.672886    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:55:08.673084 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:08.672936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:55:08.674340 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:08.674311    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:55:08.697776 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:08.697726    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podStartSLOduration=1.58527616 podStartE2EDuration="17.69767218s" totalImagesPullingTime="11.393775604s" totalInitContainerRuntime="4.718620416s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:54:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:55:08.697348368 +0000 UTC m=+2667.341317854" watchObservedRunningTime="2026-07-28 17:55:08.69767218 +0000 UTC m=+2667.341641670"
Jul 28 17:55:09.677201 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:09.677162    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:55:14.682870 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:14.682755    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:55:14.683393 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:14.683361    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:55:24.684322 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:24.684275    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:55:34.683549 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:34.683465    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:55:44.683386 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:44.683348    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:55:54.684354 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:55:54.684323    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:56:02.696873 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:02.696840    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"]
Jul 28 17:56:02.697724 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:02.697158    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container" containerID="cri-o://9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127" gracePeriod=30
Jul 28 17:56:02.697724 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:02.697192    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kube-rbac-proxy" containerID="cri-o://1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb" gracePeriod=30
Jul 28 17:56:02.894133 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:02.894097    2577 generic.go:356] "Generic (PLEG): container finished" podID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerID="1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb" exitCode=2
Jul 28 17:56:02.894337 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:02.894149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" event={"ID":"9850a394-c323-4719-8fd2-a7bae112e4cd","Type":"ContainerDied","Data":"1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb"}
Jul 28 17:56:04.678057 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:04.678018    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 28 17:56:04.683809 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:04.683751    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:56:05.542983 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.542961    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:56:05.653452 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.653410    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnrz2\" (UniqueName: \"kubernetes.io/projected/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-api-access-bnrz2\") pod \"9850a394-c323-4719-8fd2-a7bae112e4cd\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") "
Jul 28 17:56:05.653452 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.653460    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-rbac-proxy-sar-config\") pod \"9850a394-c323-4719-8fd2-a7bae112e4cd\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") "
Jul 28 17:56:05.653670 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.653491    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9850a394-c323-4719-8fd2-a7bae112e4cd-proxy-tls\") pod \"9850a394-c323-4719-8fd2-a7bae112e4cd\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") "
Jul 28 17:56:05.653670 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.653541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-cabundle-cert\") pod \"9850a394-c323-4719-8fd2-a7bae112e4cd\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") "
Jul 28 17:56:05.653670 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.653563    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9850a394-c323-4719-8fd2-a7bae112e4cd-kserve-provision-location\") pod \"9850a394-c323-4719-8fd2-a7bae112e4cd\" (UID: \"9850a394-c323-4719-8fd2-a7bae112e4cd\") "
Jul 28 17:56:05.653915 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.653891    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9850a394-c323-4719-8fd2-a7bae112e4cd" (UID: "9850a394-c323-4719-8fd2-a7bae112e4cd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:56:05.654006 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.653927    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9850a394-c323-4719-8fd2-a7bae112e4cd" (UID: "9850a394-c323-4719-8fd2-a7bae112e4cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:56:05.655695 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.655670    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-api-access-bnrz2" (OuterVolumeSpecName: "kube-api-access-bnrz2") pod "9850a394-c323-4719-8fd2-a7bae112e4cd" (UID: "9850a394-c323-4719-8fd2-a7bae112e4cd"). InnerVolumeSpecName "kube-api-access-bnrz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:56:05.655695 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.655685    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9850a394-c323-4719-8fd2-a7bae112e4cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9850a394-c323-4719-8fd2-a7bae112e4cd" (UID: "9850a394-c323-4719-8fd2-a7bae112e4cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:56:05.662018 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.661986    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9850a394-c323-4719-8fd2-a7bae112e4cd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9850a394-c323-4719-8fd2-a7bae112e4cd" (UID: "9850a394-c323-4719-8fd2-a7bae112e4cd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:05.754198 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.754162    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:56:05.754198 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.754192    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9850a394-c323-4719-8fd2-a7bae112e4cd-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:56:05.754198 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.754204    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnrz2\" (UniqueName: \"kubernetes.io/projected/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-api-access-bnrz2\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:56:05.754198 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.754212    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9850a394-c323-4719-8fd2-a7bae112e4cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:56:05.754728 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.754222    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9850a394-c323-4719-8fd2-a7bae112e4cd-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 17:56:05.909553 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.909466    2577 generic.go:356] "Generic (PLEG): container finished" podID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerID="9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127" exitCode=0
Jul 28 17:56:05.909553 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.909519    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"
Jul 28 17:56:05.909553 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.909535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" event={"ID":"9850a394-c323-4719-8fd2-a7bae112e4cd","Type":"ContainerDied","Data":"9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127"}
Jul 28 17:56:05.909767 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.909598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl" event={"ID":"9850a394-c323-4719-8fd2-a7bae112e4cd","Type":"ContainerDied","Data":"d9e505f6d978493c5d11b063171621d0e5aa8721a87ea2a89456bd57ddfd576c"}
Jul 28 17:56:05.909767 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.909627    2577 scope.go:117] "RemoveContainer" containerID="1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb"
Jul 28 17:56:05.918153 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.918136    2577 scope.go:117] "RemoveContainer" containerID="9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127"
Jul 28 17:56:05.925670 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.925650    2577 scope.go:117] "RemoveContainer" containerID="d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428"
Jul 28 17:56:05.931386 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.931365    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"]
Jul 28 17:56:05.933458 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.933440    2577 scope.go:117] "RemoveContainer" containerID="1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb"
Jul 28 17:56:05.933957 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:56:05.933930    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb\": container with ID starting with 1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb not found: ID does not exist" containerID="1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb"
Jul 28 17:56:05.934048 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.933968    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb"} err="failed to get container status \"1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb\": rpc error: code = NotFound desc = could not find container \"1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb\": container with ID starting with 1fd162af55c3c466c27bae2e53dc81ab4837c04766887beec53023436e788ebb not found: ID does not exist"
Jul 28 17:56:05.934048 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.933994    2577 scope.go:117] "RemoveContainer" containerID="9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127"
Jul 28 17:56:05.934261 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:56:05.934227    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127\": container with ID starting with 9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127 not found: ID does not exist" containerID="9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127"
Jul 28 17:56:05.934320 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.934270    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127"} err="failed to get container status \"9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127\": rpc error: code = NotFound desc = could not find container \"9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127\": container with ID starting with 9a00ee47207360ffad6d6b95b2ea62323e3fc6a9bf79a48954585962e6fe5127 not found: ID does not exist"
Jul 28 17:56:05.934320 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.934289    2577 scope.go:117] "RemoveContainer" containerID="d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428"
Jul 28 17:56:05.934525 ip-10-0-139-190 kubenswrapper[2577]: E0728 17:56:05.934503    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428\": container with ID starting with d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428 not found: ID does not exist" containerID="d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428"
Jul 28 17:56:05.934592 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.934535    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428"} err="failed to get container status \"d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428\": rpc error: code = NotFound desc = could not find container \"d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428\": container with ID starting with d0a4d914629aa75d80d8ab58283ee87b5793627699e1643bad092760957bf428 not found: ID does not exist"
Jul 28 17:56:05.935666 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:05.935651    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-lqdfl"]
Jul 28 17:56:07.909268 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:07.905908    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" path="/var/lib/kubelet/pods/9850a394-c323-4719-8fd2-a7bae112e4cd/volumes"
Jul 28 17:56:15.942064 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:15.942031    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 17:56:15.944227 ip-10-0-139-190 kubenswrapper[2577]: I0728 17:56:15.944203    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:01:15.969850 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:01:15.969814    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:01:15.973662 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:01:15.973642    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:04:25.406963 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.406929    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"]
Jul 28 18:04:25.407449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.407228    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kube-rbac-proxy"
Jul 28 18:04:25.407449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.407237    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kube-rbac-proxy"
Jul 28 18:04:25.407449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.407272    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="storage-initializer"
Jul 28 18:04:25.407449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.407279    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="storage-initializer"
Jul 28 18:04:25.407449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.407290    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container"
Jul 28 18:04:25.407449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.407296    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container"
Jul 28 18:04:25.407449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.407341    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kserve-container"
Jul 28 18:04:25.407449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.407351    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9850a394-c323-4719-8fd2-a7bae112e4cd" containerName="kube-rbac-proxy"
Jul 28 18:04:25.410462 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.410444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.413129 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.413108    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-de1b83-kube-rbac-proxy-sar-config\""
Jul 28 18:04:25.413474 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.413459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-de1b83-dockercfg-p26vw\""
Jul 28 18:04:25.413550 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.413533    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:04:25.414380 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.414360    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-de1b83\""
Jul 28 18:04:25.414380 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.414377    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-de1b83-predictor-serving-cert\""
Jul 28 18:04:25.414536 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.414380    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:04:25.414536 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.414410    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:04:25.420494 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.420474    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"]
Jul 28 18:04:25.540683 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.540650    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.540683 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.540685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061c798-807a-4539-9d5f-5f1dd6289461-kserve-provision-location\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.540914 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.540725    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061c798-807a-4539-9d5f-5f1dd6289461-proxy-tls\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.540914 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.540791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x96q\" (UniqueName: \"kubernetes.io/projected/0061c798-807a-4539-9d5f-5f1dd6289461-kube-api-access-4x96q\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.540914 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.540856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-cabundle-cert\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.641908 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.641875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x96q\" (UniqueName: \"kubernetes.io/projected/0061c798-807a-4539-9d5f-5f1dd6289461-kube-api-access-4x96q\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.642094 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.641920    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-cabundle-cert\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.642094 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.641970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.642094 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.641989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061c798-807a-4539-9d5f-5f1dd6289461-kserve-provision-location\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.642094 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.642025    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061c798-807a-4539-9d5f-5f1dd6289461-proxy-tls\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.642565 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.642540    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061c798-807a-4539-9d5f-5f1dd6289461-kserve-provision-location\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.642745 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.642723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-cabundle-cert\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.642829 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.642805    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.644554 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.644532    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061c798-807a-4539-9d5f-5f1dd6289461-proxy-tls\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.651915 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.651888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x96q\" (UniqueName: \"kubernetes.io/projected/0061c798-807a-4539-9d5f-5f1dd6289461-kube-api-access-4x96q\") pod \"isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") " pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.721301 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.721187    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:25.855995 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.855970    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"]
Jul 28 18:04:25.884162 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:25.884133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m" event={"ID":"0061c798-807a-4539-9d5f-5f1dd6289461","Type":"ContainerStarted","Data":"362a16e39375ac7a4784a4218d0218244b9aae8c92cceb6265822e13dda5831d"}
Jul 28 18:04:26.890182 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:26.890140    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m" event={"ID":"0061c798-807a-4539-9d5f-5f1dd6289461","Type":"ContainerStarted","Data":"105059798f66d53a9c66574b115b707f93bfd16359f93c4a66703147aff44de3"}
Jul 28 18:04:32.916787 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:32.916762    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_0061c798-807a-4539-9d5f-5f1dd6289461/storage-initializer/0.log"
Jul 28 18:04:32.917150 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:32.916798    2577 generic.go:356] "Generic (PLEG): container finished" podID="0061c798-807a-4539-9d5f-5f1dd6289461" containerID="105059798f66d53a9c66574b115b707f93bfd16359f93c4a66703147aff44de3" exitCode=1
Jul 28 18:04:32.917150 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:32.916878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m" event={"ID":"0061c798-807a-4539-9d5f-5f1dd6289461","Type":"ContainerDied","Data":"105059798f66d53a9c66574b115b707f93bfd16359f93c4a66703147aff44de3"}
Jul 28 18:04:33.922355 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:33.922333    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_0061c798-807a-4539-9d5f-5f1dd6289461/storage-initializer/0.log"
Jul 28 18:04:33.922744 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:33.922432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m" event={"ID":"0061c798-807a-4539-9d5f-5f1dd6289461","Type":"ContainerStarted","Data":"ebbae5648529740d5e8fc8b41b39d1122f80301f1be24472132b26ccb0e42c8d"}
Jul 28 18:04:36.936006 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:36.935979    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_0061c798-807a-4539-9d5f-5f1dd6289461/storage-initializer/1.log"
Jul 28 18:04:36.936472 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:36.936345    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_0061c798-807a-4539-9d5f-5f1dd6289461/storage-initializer/0.log"
Jul 28 18:04:36.936472 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:36.936375    2577 generic.go:356] "Generic (PLEG): container finished" podID="0061c798-807a-4539-9d5f-5f1dd6289461" containerID="ebbae5648529740d5e8fc8b41b39d1122f80301f1be24472132b26ccb0e42c8d" exitCode=1
Jul 28 18:04:36.936472 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:36.936405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m" event={"ID":"0061c798-807a-4539-9d5f-5f1dd6289461","Type":"ContainerDied","Data":"ebbae5648529740d5e8fc8b41b39d1122f80301f1be24472132b26ccb0e42c8d"}
Jul 28 18:04:36.936472 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:36.936451    2577 scope.go:117] "RemoveContainer" containerID="105059798f66d53a9c66574b115b707f93bfd16359f93c4a66703147aff44de3"
Jul 28 18:04:36.936758 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:36.936738    2577 scope.go:117] "RemoveContainer" containerID="105059798f66d53a9c66574b115b707f93bfd16359f93c4a66703147aff44de3"
Jul 28 18:04:36.946876 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:04:36.946855    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_kserve-ci-e2e-test(0061c798-807a-4539-9d5f-5f1dd6289461)\"" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m" podUID="0061c798-807a-4539-9d5f-5f1dd6289461"
Jul 28 18:04:36.947080 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:04:36.947055    2577 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_kserve-ci-e2e-test_0061c798-807a-4539-9d5f-5f1dd6289461_0 in pod sandbox 362a16e39375ac7a4784a4218d0218244b9aae8c92cceb6265822e13dda5831d from index: no such id: '105059798f66d53a9c66574b115b707f93bfd16359f93c4a66703147aff44de3'" containerID="105059798f66d53a9c66574b115b707f93bfd16359f93c4a66703147aff44de3"
Jul 28 18:04:36.947137 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:36.947089    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"105059798f66d53a9c66574b115b707f93bfd16359f93c4a66703147aff44de3"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_kserve-ci-e2e-test_0061c798-807a-4539-9d5f-5f1dd6289461_0 in pod sandbox 362a16e39375ac7a4784a4218d0218244b9aae8c92cceb6265822e13dda5831d from index: no such id: '105059798f66d53a9c66574b115b707f93bfd16359f93c4a66703147aff44de3'"
Jul 28 18:04:37.941772 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:37.941746    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_0061c798-807a-4539-9d5f-5f1dd6289461/storage-initializer/1.log"
Jul 28 18:04:43.425041 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.425005    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"]
Jul 28 18:04:43.559028 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.559006    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_0061c798-807a-4539-9d5f-5f1dd6289461/storage-initializer/1.log"
Jul 28 18:04:43.562566 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.562551    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:43.683658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.683571    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x96q\" (UniqueName: \"kubernetes.io/projected/0061c798-807a-4539-9d5f-5f1dd6289461-kube-api-access-4x96q\") pod \"0061c798-807a-4539-9d5f-5f1dd6289461\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") "
Jul 28 18:04:43.683658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.683625    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-kube-rbac-proxy-sar-config\") pod \"0061c798-807a-4539-9d5f-5f1dd6289461\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") "
Jul 28 18:04:43.683658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.683644    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061c798-807a-4539-9d5f-5f1dd6289461-proxy-tls\") pod \"0061c798-807a-4539-9d5f-5f1dd6289461\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") "
Jul 28 18:04:43.683658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.683668    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-cabundle-cert\") pod \"0061c798-807a-4539-9d5f-5f1dd6289461\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") "
Jul 28 18:04:43.683981 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.683701    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061c798-807a-4539-9d5f-5f1dd6289461-kserve-provision-location\") pod \"0061c798-807a-4539-9d5f-5f1dd6289461\" (UID: \"0061c798-807a-4539-9d5f-5f1dd6289461\") "
Jul 28 18:04:43.684036 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.683991    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0061c798-807a-4539-9d5f-5f1dd6289461-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0061c798-807a-4539-9d5f-5f1dd6289461" (UID: "0061c798-807a-4539-9d5f-5f1dd6289461"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:04:43.684077 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.684027    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0061c798-807a-4539-9d5f-5f1dd6289461" (UID: "0061c798-807a-4539-9d5f-5f1dd6289461"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:43.684077 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.684042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0061c798-807a-4539-9d5f-5f1dd6289461" (UID: "0061c798-807a-4539-9d5f-5f1dd6289461"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:04:43.685708 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.685672    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0061c798-807a-4539-9d5f-5f1dd6289461-kube-api-access-4x96q" (OuterVolumeSpecName: "kube-api-access-4x96q") pod "0061c798-807a-4539-9d5f-5f1dd6289461" (UID: "0061c798-807a-4539-9d5f-5f1dd6289461"). InnerVolumeSpecName "kube-api-access-4x96q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:04:43.685800 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.685743    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0061c798-807a-4539-9d5f-5f1dd6289461-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0061c798-807a-4539-9d5f-5f1dd6289461" (UID: "0061c798-807a-4539-9d5f-5f1dd6289461"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:04:43.785018 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.784984    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x96q\" (UniqueName: \"kubernetes.io/projected/0061c798-807a-4539-9d5f-5f1dd6289461-kube-api-access-4x96q\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:04:43.785018 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.785017    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:04:43.785018 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.785027    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0061c798-807a-4539-9d5f-5f1dd6289461-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:04:43.785018 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.785037    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0061c798-807a-4539-9d5f-5f1dd6289461-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:04:43.785272 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.785049    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0061c798-807a-4539-9d5f-5f1dd6289461-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:04:43.965231 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.965163    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m_0061c798-807a-4539-9d5f-5f1dd6289461/storage-initializer/1.log"
Jul 28 18:04:43.965231 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.965212    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m" event={"ID":"0061c798-807a-4539-9d5f-5f1dd6289461","Type":"ContainerDied","Data":"362a16e39375ac7a4784a4218d0218244b9aae8c92cceb6265822e13dda5831d"}
Jul 28 18:04:43.965412 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.965299    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"
Jul 28 18:04:43.965412 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:43.965336    2577 scope.go:117] "RemoveContainer" containerID="ebbae5648529740d5e8fc8b41b39d1122f80301f1be24472132b26ccb0e42c8d"
Jul 28 18:04:44.000594 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:44.000566    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"]
Jul 28 18:04:44.005202 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:44.005176    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-de1b83-predictor-6d7b575cc4-bft7m"]
Jul 28 18:04:45.904329 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:04:45.904297    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0061c798-807a-4539-9d5f-5f1dd6289461" path="/var/lib/kubelet/pods/0061c798-807a-4539-9d5f-5f1dd6289461/volumes"
Jul 28 18:06:15.995202 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:06:15.995167    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:06:15.999898 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:06:15.999877    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:08:14.544600 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.544558    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"]
Jul 28 18:08:14.545026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.544916    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0061c798-807a-4539-9d5f-5f1dd6289461" containerName="storage-initializer"
Jul 28 18:08:14.545026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.544929    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0061c798-807a-4539-9d5f-5f1dd6289461" containerName="storage-initializer"
Jul 28 18:08:14.545026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.544942    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0061c798-807a-4539-9d5f-5f1dd6289461" containerName="storage-initializer"
Jul 28 18:08:14.545026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.544947    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0061c798-807a-4539-9d5f-5f1dd6289461" containerName="storage-initializer"
Jul 28 18:08:14.545026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.545013    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0061c798-807a-4539-9d5f-5f1dd6289461" containerName="storage-initializer"
Jul 28 18:08:14.545026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.545023    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0061c798-807a-4539-9d5f-5f1dd6289461" containerName="storage-initializer"
Jul 28 18:08:14.548189 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.548170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.550865 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.550836    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 28 18:08:14.550994 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.550968    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:08:14.550994 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.550987    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:08:14.551090 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.551079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:08:14.552022 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.551998    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 28 18:08:14.552139 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.552030    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:08:14.552139 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.552084    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:08:14.557782 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.557762    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"]
Jul 28 18:08:14.660536 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.660492    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4zdm\" (UniqueName: \"kubernetes.io/projected/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-api-access-n4zdm\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.660536 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.660540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.660786 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.660598    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.660786 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.660684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.660786 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.660715    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.761785 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.761745    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.761785 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.761788    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.762030 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.761838    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4zdm\" (UniqueName: \"kubernetes.io/projected/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-api-access-n4zdm\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.762030 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.761860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.762030 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.761889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.762030 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:08:14.761916    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 28 18:08:14.762030 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:08:14.761999    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-proxy-tls podName:112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34 nodeName:}" failed. No retries permitted until 2026-07-28 18:08:15.261976831 +0000 UTC m=+3453.905946310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" (UID: "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 28 18:08:14.762353 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.762330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.762638 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.762617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.762690 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.762625    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:14.771346 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:14.771322    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4zdm\" (UniqueName: \"kubernetes.io/projected/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-api-access-n4zdm\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:15.266869 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:15.266831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:15.269204 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:15.269184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:15.459374 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:15.459338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:15.590324 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:15.590300    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"]
Jul 28 18:08:15.592357 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:08:15.592328    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112fc7ae_a6c3_4a63_b3a8_bbfaf7561f34.slice/crio-4f1443175191847bdcb08f0227910171514002cf682c03913e8ad8bf5a8e0c77 WatchSource:0}: Error finding container 4f1443175191847bdcb08f0227910171514002cf682c03913e8ad8bf5a8e0c77: Status 404 returned error can't find the container with id 4f1443175191847bdcb08f0227910171514002cf682c03913e8ad8bf5a8e0c77
Jul 28 18:08:15.816453 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:15.816365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" event={"ID":"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34","Type":"ContainerStarted","Data":"687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8"}
Jul 28 18:08:15.816453 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:15.816437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" event={"ID":"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34","Type":"ContainerStarted","Data":"4f1443175191847bdcb08f0227910171514002cf682c03913e8ad8bf5a8e0c77"}
Jul 28 18:08:19.833945 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:19.833857    2577 generic.go:356] "Generic (PLEG): container finished" podID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerID="687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8" exitCode=0
Jul 28 18:08:19.833945 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:19.833916    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" event={"ID":"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34","Type":"ContainerDied","Data":"687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8"}
Jul 28 18:08:19.835181 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:19.835162    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:08:42.952058 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:42.952021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" event={"ID":"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34","Type":"ContainerStarted","Data":"404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28"}
Jul 28 18:08:42.952613 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:42.952093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" event={"ID":"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34","Type":"ContainerStarted","Data":"e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54"}
Jul 28 18:08:42.952613 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:42.952346    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:42.976072 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:42.976024    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podStartSLOduration=2.494409538 podStartE2EDuration="28.976007496s" totalImagesPullingTime="22.552398463s" totalInitContainerRuntime="3.929199495s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:08:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:08:42.975396765 +0000 UTC m=+3481.619366255" watchObservedRunningTime="2026-07-28 18:08:42.976007496 +0000 UTC m=+3481.619976987"
Jul 28 18:08:43.956671 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:43.956640    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:43.957846 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:43.957813    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 18:08:44.961092 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:44.961055    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 18:08:49.966086 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:49.966057    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:08:49.966556 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:49.966531    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 18:08:59.967070 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:08:59.967032    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 18:09:09.966904 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:09:09.966819    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 18:09:19.967004 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:09:19.966966    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 18:09:29.966509 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:09:29.966466    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 18:09:39.967489 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:09:39.967447    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 18:09:49.967496 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:09:49.967451    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 18:09:59.967909 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:09:59.967882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:10:04.717099 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:04.717060    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"]
Jul 28 18:10:04.717629 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:04.717413    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container" containerID="cri-o://e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54" gracePeriod=30
Jul 28 18:10:04.717629 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:04.717457    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kube-rbac-proxy" containerID="cri-o://404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28" gracePeriod=30
Jul 28 18:10:04.961422 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:04.961379    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 28 18:10:05.286096 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:05.286065    2577 generic.go:356] "Generic (PLEG): container finished" podID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerID="404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28" exitCode=2
Jul 28 18:10:05.286298 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:05.286136    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" event={"ID":"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34","Type":"ContainerDied","Data":"404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28"}
Jul 28 18:10:09.859870 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.859845    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:10:09.917394 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.917356    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"]
Jul 28 18:10:09.917743 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.917725    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kube-rbac-proxy"
Jul 28 18:10:09.917824 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.917745    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kube-rbac-proxy"
Jul 28 18:10:09.917824 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.917764    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container"
Jul 28 18:10:09.917824 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.917773    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container"
Jul 28 18:10:09.917824 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.917784    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="storage-initializer"
Jul 28 18:10:09.917824 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.917793    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="storage-initializer"
Jul 28 18:10:09.918082 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.917881    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kube-rbac-proxy"
Jul 28 18:10:09.918082 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.917896    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerName="kserve-container"
Jul 28 18:10:09.920042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.920019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:09.922832 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.922814    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:10:09.922932 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.922891    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 28 18:10:09.932268 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.932229    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"]
Jul 28 18:10:09.951210 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.951158    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kserve-provision-location\") pod \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") "
Jul 28 18:10:09.951443 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.951291    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-proxy-tls\") pod \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") "
Jul 28 18:10:09.951443 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.951326    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-rbac-proxy-sar-config\") pod \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") "
Jul 28 18:10:09.951443 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.951382    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-cabundle-cert\") pod \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") "
Jul 28 18:10:09.951443 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.951408    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4zdm\" (UniqueName: \"kubernetes.io/projected/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-api-access-n4zdm\") pod \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\" (UID: \"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34\") "
Jul 28 18:10:09.951702 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.951514    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" (UID: "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:10:09.951702 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.951623    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:10:09.951963 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.951942    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" (UID: "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:10:09.952264 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.952211    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" (UID: "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:10:09.953666 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.953641    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-api-access-n4zdm" (OuterVolumeSpecName: "kube-api-access-n4zdm") pod "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" (UID: "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34"). InnerVolumeSpecName "kube-api-access-n4zdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:10:09.953762 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:09.953744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" (UID: "112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:10:10.052356 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.052324    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.052556 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.052470    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32effda0-10f6-4f1c-ab92-501dbbc73677-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.052556 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.052521    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32effda0-10f6-4f1c-ab92-501dbbc73677-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.052658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.052563    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6kcr\" (UniqueName: \"kubernetes.io/projected/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-api-access-h6kcr\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.052801 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.052772    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.052872 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.052860    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:10:10.052914 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.052876    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4zdm\" (UniqueName: \"kubernetes.io/projected/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-api-access-n4zdm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:10:10.052914 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.052891    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:10:10.052914 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.052900    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:10:10.153278 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.153220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.153474 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.153315    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32effda0-10f6-4f1c-ab92-501dbbc73677-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.153474 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.153352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32effda0-10f6-4f1c-ab92-501dbbc73677-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.153474 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.153392    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6kcr\" (UniqueName: \"kubernetes.io/projected/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-api-access-h6kcr\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.153474 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.153427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.153758 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.153733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32effda0-10f6-4f1c-ab92-501dbbc73677-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.154084 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.154061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.154131 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.154074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.155718 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.155695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32effda0-10f6-4f1c-ab92-501dbbc73677-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.163479 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.163460    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6kcr\" (UniqueName: \"kubernetes.io/projected/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-api-access-h6kcr\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.229822 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.229788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:10.308823 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.308795    2577 generic.go:356] "Generic (PLEG): container finished" podID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" containerID="e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54" exitCode=0
Jul 28 18:10:10.308967 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.308837    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" event={"ID":"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34","Type":"ContainerDied","Data":"e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54"}
Jul 28 18:10:10.308967 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.308866    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"
Jul 28 18:10:10.308967 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.308907    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg" event={"ID":"112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34","Type":"ContainerDied","Data":"4f1443175191847bdcb08f0227910171514002cf682c03913e8ad8bf5a8e0c77"}
Jul 28 18:10:10.308967 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.308946    2577 scope.go:117] "RemoveContainer" containerID="404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28"
Jul 28 18:10:10.319927 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.319906    2577 scope.go:117] "RemoveContainer" containerID="e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54"
Jul 28 18:10:10.330147 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.330121    2577 scope.go:117] "RemoveContainer" containerID="687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8"
Jul 28 18:10:10.339056 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.339034    2577 scope.go:117] "RemoveContainer" containerID="404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28"
Jul 28 18:10:10.339410 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:10:10.339387    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28\": container with ID starting with 404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28 not found: ID does not exist" containerID="404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28"
Jul 28 18:10:10.339522 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.339433    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28"} err="failed to get container status \"404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28\": rpc error: code = NotFound desc = could not find container \"404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28\": container with ID starting with 404ad53b97b236ee295e1e285dda235998547ccda22417f9475f07af9a9c4a28 not found: ID does not exist"
Jul 28 18:10:10.339522 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.339461    2577 scope.go:117] "RemoveContainer" containerID="e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54"
Jul 28 18:10:10.339884 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.339862    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"]
Jul 28 18:10:10.339971 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:10:10.339868    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54\": container with ID starting with e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54 not found: ID does not exist" containerID="e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54"
Jul 28 18:10:10.339971 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.339912    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54"} err="failed to get container status \"e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54\": rpc error: code = NotFound desc = could not find container \"e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54\": container with ID starting with e531a71af63df38e2693126c8e370acbb8910cf6bbc8a25a3b5758d31da9cc54 not found: ID does not exist"
Jul 28 18:10:10.339971 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.339928    2577 scope.go:117] "RemoveContainer" containerID="687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8"
Jul 28 18:10:10.340187 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:10:10.340169    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8\": container with ID starting with 687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8 not found: ID does not exist" containerID="687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8"
Jul 28 18:10:10.340237 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.340193    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8"} err="failed to get container status \"687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8\": rpc error: code = NotFound desc = could not find container \"687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8\": container with ID starting with 687358ae7df761c5726767e76d7696f8c1afbf5b24ae3ad66e9b45b64c5a03d8 not found: ID does not exist"
Jul 28 18:10:10.344682 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.344646    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-hlctg"]
Jul 28 18:10:10.368138 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:10.368115    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"]
Jul 28 18:10:10.370481 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:10:10.370450    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32effda0_10f6_4f1c_ab92_501dbbc73677.slice/crio-7fb85a16fa6cd634466a98ff3b9f54e571468278416cc0ecd99c4828c16e5db2 WatchSource:0}: Error finding container 7fb85a16fa6cd634466a98ff3b9f54e571468278416cc0ecd99c4828c16e5db2: Status 404 returned error can't find the container with id 7fb85a16fa6cd634466a98ff3b9f54e571468278416cc0ecd99c4828c16e5db2
Jul 28 18:10:11.315608 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:11.315573    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" event={"ID":"32effda0-10f6-4f1c-ab92-501dbbc73677","Type":"ContainerStarted","Data":"0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850"}
Jul 28 18:10:11.316007 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:11.315628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" event={"ID":"32effda0-10f6-4f1c-ab92-501dbbc73677","Type":"ContainerStarted","Data":"7fb85a16fa6cd634466a98ff3b9f54e571468278416cc0ecd99c4828c16e5db2"}
Jul 28 18:10:11.904853 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:11.904821    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34" path="/var/lib/kubelet/pods/112fc7ae-a6c3-4a63-b3a8-bbfaf7561f34/volumes"
Jul 28 18:10:15.333548 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:15.333511    2577 generic.go:356] "Generic (PLEG): container finished" podID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerID="0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850" exitCode=0
Jul 28 18:10:15.333933 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:15.333586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" event={"ID":"32effda0-10f6-4f1c-ab92-501dbbc73677","Type":"ContainerDied","Data":"0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850"}
Jul 28 18:10:16.340543 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:16.340499    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" event={"ID":"32effda0-10f6-4f1c-ab92-501dbbc73677","Type":"ContainerStarted","Data":"4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f"}
Jul 28 18:10:16.340949 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:16.340623    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" event={"ID":"32effda0-10f6-4f1c-ab92-501dbbc73677","Type":"ContainerStarted","Data":"b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588"}
Jul 28 18:10:16.340949 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:16.340890    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:16.361644 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:16.361593    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" podStartSLOduration=3.4208476660000002 podStartE2EDuration="7.361578298s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.940730632s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:10:16.361439666 +0000 UTC m=+3575.005409158" watchObservedRunningTime="2026-07-28 18:10:16.361578298 +0000 UTC m=+3575.005547789"
Jul 28 18:10:17.345226 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:17.345195    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:23.355960 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:23.355935    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:10:53.357448 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:10:53.357410    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 18:11:03.356499 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:03.356462    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 18:11:13.357377 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:13.357339    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 18:11:16.021055 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:16.021028    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:11:16.026989 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:16.026966    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:11:23.357344 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:23.357302    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 18:11:33.439865 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:33.439834    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:11:40.952530 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:40.952494    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"]
Jul 28 18:11:40.952945 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:40.952762    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kserve-container" containerID="cri-o://b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588" gracePeriod=30
Jul 28 18:11:40.952945 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:40.952791    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kube-rbac-proxy" containerID="cri-o://4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f" gracePeriod=30
Jul 28 18:11:41.697934 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:41.697894    2577 generic.go:356] "Generic (PLEG): container finished" podID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerID="4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f" exitCode=2
Jul 28 18:11:41.698101 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:41.697967    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" event={"ID":"32effda0-10f6-4f1c-ab92-501dbbc73677","Type":"ContainerDied","Data":"4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f"}
Jul 28 18:11:43.351024 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:43.350983    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 28 18:11:43.357289 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:43.357263    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 18:11:45.705642 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.705618    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:11:45.714724 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.714692    2577 generic.go:356] "Generic (PLEG): container finished" podID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerID="b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588" exitCode=0
Jul 28 18:11:45.714857 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.714761    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"
Jul 28 18:11:45.714857 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.714759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" event={"ID":"32effda0-10f6-4f1c-ab92-501dbbc73677","Type":"ContainerDied","Data":"b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588"}
Jul 28 18:11:45.714977 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.714884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph" event={"ID":"32effda0-10f6-4f1c-ab92-501dbbc73677","Type":"ContainerDied","Data":"7fb85a16fa6cd634466a98ff3b9f54e571468278416cc0ecd99c4828c16e5db2"}
Jul 28 18:11:45.714977 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.714911    2577 scope.go:117] "RemoveContainer" containerID="4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f"
Jul 28 18:11:45.722667 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.722649    2577 scope.go:117] "RemoveContainer" containerID="b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588"
Jul 28 18:11:45.730077 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.730049    2577 scope.go:117] "RemoveContainer" containerID="0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850"
Jul 28 18:11:45.739387 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.739369    2577 scope.go:117] "RemoveContainer" containerID="4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f"
Jul 28 18:11:45.739624 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:11:45.739606    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f\": container with ID starting with 4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f not found: ID does not exist" containerID="4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f"
Jul 28 18:11:45.739689 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.739631    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f"} err="failed to get container status \"4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f\": rpc error: code = NotFound desc = could not find container \"4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f\": container with ID starting with 4f8740787ef13a306bc0ba3dbb3b4ada0aef6bc969f1b9e5512bc45a8387491f not found: ID does not exist"
Jul 28 18:11:45.739689 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.739647    2577 scope.go:117] "RemoveContainer" containerID="b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588"
Jul 28 18:11:45.739871 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:11:45.739856    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588\": container with ID starting with b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588 not found: ID does not exist" containerID="b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588"
Jul 28 18:11:45.739917 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.739874    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588"} err="failed to get container status \"b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588\": rpc error: code = NotFound desc = could not find container \"b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588\": container with ID starting with b9157ee9e85eac9922dbc8eb0444fc1c273d7b8fd5e1b95df10cd736c1aca588 not found: ID does not exist"
Jul 28 18:11:45.739917 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.739885    2577 scope.go:117] "RemoveContainer" containerID="0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850"
Jul 28 18:11:45.740092 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:11:45.740077    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850\": container with ID starting with 0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850 not found: ID does not exist" containerID="0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850"
Jul 28 18:11:45.740145 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.740095    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850"} err="failed to get container status \"0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850\": rpc error: code = NotFound desc = could not find container \"0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850\": container with ID starting with 0ac18f4802dc38a0f33d1c3af7b67d81400f1fc2da5ec8c4fcb6b16b944e5850 not found: ID does not exist"
Jul 28 18:11:45.851590 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.851502    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-cabundle-cert\") pod \"32effda0-10f6-4f1c-ab92-501dbbc73677\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") "
Jul 28 18:11:45.851590 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.851551    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32effda0-10f6-4f1c-ab92-501dbbc73677-proxy-tls\") pod \"32effda0-10f6-4f1c-ab92-501dbbc73677\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") "
Jul 28 18:11:45.851590 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.851582    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6kcr\" (UniqueName: \"kubernetes.io/projected/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-api-access-h6kcr\") pod \"32effda0-10f6-4f1c-ab92-501dbbc73677\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") "
Jul 28 18:11:45.851842 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.851643    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-rbac-proxy-sar-config\") pod \"32effda0-10f6-4f1c-ab92-501dbbc73677\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") "
Jul 28 18:11:45.851842 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.851670    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32effda0-10f6-4f1c-ab92-501dbbc73677-kserve-provision-location\") pod \"32effda0-10f6-4f1c-ab92-501dbbc73677\" (UID: \"32effda0-10f6-4f1c-ab92-501dbbc73677\") "
Jul 28 18:11:45.852042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.852014    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "32effda0-10f6-4f1c-ab92-501dbbc73677" (UID: "32effda0-10f6-4f1c-ab92-501dbbc73677"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:11:45.852163 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.852078    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32effda0-10f6-4f1c-ab92-501dbbc73677-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32effda0-10f6-4f1c-ab92-501dbbc73677" (UID: "32effda0-10f6-4f1c-ab92-501dbbc73677"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:11:45.852163 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.852078    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "32effda0-10f6-4f1c-ab92-501dbbc73677" (UID: "32effda0-10f6-4f1c-ab92-501dbbc73677"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:11:45.853760 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.853738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32effda0-10f6-4f1c-ab92-501dbbc73677-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "32effda0-10f6-4f1c-ab92-501dbbc73677" (UID: "32effda0-10f6-4f1c-ab92-501dbbc73677"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:11:45.853852 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.853760    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-api-access-h6kcr" (OuterVolumeSpecName: "kube-api-access-h6kcr") pod "32effda0-10f6-4f1c-ab92-501dbbc73677" (UID: "32effda0-10f6-4f1c-ab92-501dbbc73677"). InnerVolumeSpecName "kube-api-access-h6kcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:11:45.953015 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.952980    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6kcr\" (UniqueName: \"kubernetes.io/projected/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-api-access-h6kcr\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:11:45.953015 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.953004    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:11:45.953015 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.953014    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32effda0-10f6-4f1c-ab92-501dbbc73677-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:11:45.953015 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.953023    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/32effda0-10f6-4f1c-ab92-501dbbc73677-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:11:45.953384 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:45.953033    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/32effda0-10f6-4f1c-ab92-501dbbc73677-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:11:46.039584 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:46.039546    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"]
Jul 28 18:11:46.047065 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:46.047036    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-ccmph"]
Jul 28 18:11:47.904076 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:11:47.904038    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" path="/var/lib/kubelet/pods/32effda0-10f6-4f1c-ab92-501dbbc73677/volumes"
Jul 28 18:13:11.415548 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.415513    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"]
Jul 28 18:13:11.415970 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.415846    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kube-rbac-proxy"
Jul 28 18:13:11.415970 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.415856    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kube-rbac-proxy"
Jul 28 18:13:11.415970 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.415866    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="storage-initializer"
Jul 28 18:13:11.415970 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.415871    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="storage-initializer"
Jul 28 18:13:11.415970 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.415886    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kserve-container"
Jul 28 18:13:11.415970 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.415892    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kserve-container"
Jul 28 18:13:11.415970 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.415941    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kserve-container"
Jul 28 18:13:11.415970 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.415949    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="32effda0-10f6-4f1c-ab92-501dbbc73677" containerName="kube-rbac-proxy"
Jul 28 18:13:11.419045 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.419029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.421848 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.421826    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 28 18:13:11.422089 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.422069    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:13:11.422185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.422128    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:13:11.422363 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.422348    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:13:11.423149 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.423136    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:13:11.423266 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.423136    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 28 18:13:11.423472 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.423456    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:13:11.429533 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.429513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"]
Jul 28 18:13:11.552863 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.552827    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24780714-585d-4aa7-b287-9a981264b266-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.552863 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.552863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r875f\" (UniqueName: \"kubernetes.io/projected/24780714-585d-4aa7-b287-9a981264b266-kube-api-access-r875f\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.553081 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.552941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.553081 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.552988    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24780714-585d-4aa7-b287-9a981264b266-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.553081 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.553026    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.654468 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.654419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.654468 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.654474    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24780714-585d-4aa7-b287-9a981264b266-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.654734 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.654497    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r875f\" (UniqueName: \"kubernetes.io/projected/24780714-585d-4aa7-b287-9a981264b266-kube-api-access-r875f\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.654734 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.654539    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.654734 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.654572    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24780714-585d-4aa7-b287-9a981264b266-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.655010 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.654981    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24780714-585d-4aa7-b287-9a981264b266-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.655206 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.655184    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.655294 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.655216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.657087 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.657061    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24780714-585d-4aa7-b287-9a981264b266-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.664575 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.664550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r875f\" (UniqueName: \"kubernetes.io/projected/24780714-585d-4aa7-b287-9a981264b266-kube-api-access-r875f\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.729675 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.729591    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:11.860716 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:13:11.860687    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24780714_585d_4aa7_b287_9a981264b266.slice/crio-788efa00e5af2a8cfa4fbd4d370133526a0dd2b4b2239088c87c2eb3d5268d42 WatchSource:0}: Error finding container 788efa00e5af2a8cfa4fbd4d370133526a0dd2b4b2239088c87c2eb3d5268d42: Status 404 returned error can't find the container with id 788efa00e5af2a8cfa4fbd4d370133526a0dd2b4b2239088c87c2eb3d5268d42
Jul 28 18:13:11.861137 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:11.861121    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"]
Jul 28 18:13:12.058883 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:12.058796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" event={"ID":"24780714-585d-4aa7-b287-9a981264b266","Type":"ContainerStarted","Data":"1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97"}
Jul 28 18:13:12.058883 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:12.058865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" event={"ID":"24780714-585d-4aa7-b287-9a981264b266","Type":"ContainerStarted","Data":"788efa00e5af2a8cfa4fbd4d370133526a0dd2b4b2239088c87c2eb3d5268d42"}
Jul 28 18:13:16.076584 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:16.076553    2577 generic.go:356] "Generic (PLEG): container finished" podID="24780714-585d-4aa7-b287-9a981264b266" containerID="1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97" exitCode=0
Jul 28 18:13:16.077042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:16.076621    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" event={"ID":"24780714-585d-4aa7-b287-9a981264b266","Type":"ContainerDied","Data":"1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97"}
Jul 28 18:13:17.083051 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:17.083015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" event={"ID":"24780714-585d-4aa7-b287-9a981264b266","Type":"ContainerStarted","Data":"18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293"}
Jul 28 18:13:17.083513 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:17.083078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" event={"ID":"24780714-585d-4aa7-b287-9a981264b266","Type":"ContainerStarted","Data":"5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506"}
Jul 28 18:13:17.083513 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:17.083293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:17.083513 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:17.083340    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:17.107127 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:17.107085    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" podStartSLOduration=2.404967382 podStartE2EDuration="6.107071002s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.70210362s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:13:17.105041043 +0000 UTC m=+3755.749010544" watchObservedRunningTime="2026-07-28 18:13:17.107071002 +0000 UTC m=+3755.751040533"
Jul 28 18:13:23.093805 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:23.093779    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:13:53.094911 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:13:53.094825    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 18:14:03.094813 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:03.094771    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 18:14:13.094822 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:13.094778    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 18:14:23.094457 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:23.094421    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.34:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 18:14:30.937911 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:30.937878    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:14:31.505707 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:31.505679    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"]
Jul 28 18:14:32.392731 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:32.392687    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kserve-container" containerID="cri-o://5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506" gracePeriod=30
Jul 28 18:14:32.393162 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:32.392725    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kube-rbac-proxy" containerID="cri-o://18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293" gracePeriod=30
Jul 28 18:14:33.089473 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:33.089428    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 28 18:14:33.398289 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:33.398257    2577 generic.go:356] "Generic (PLEG): container finished" podID="24780714-585d-4aa7-b287-9a981264b266" containerID="18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293" exitCode=2
Jul 28 18:14:33.398650 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:33.398307    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" event={"ID":"24780714-585d-4aa7-b287-9a981264b266","Type":"ContainerDied","Data":"18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293"}
Jul 28 18:14:36.936208 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:36.936186    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:14:37.118140 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.118040    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r875f\" (UniqueName: \"kubernetes.io/projected/24780714-585d-4aa7-b287-9a981264b266-kube-api-access-r875f\") pod \"24780714-585d-4aa7-b287-9a981264b266\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") "
Jul 28 18:14:37.118140 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.118091    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-kube-rbac-proxy-sar-config\") pod \"24780714-585d-4aa7-b287-9a981264b266\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") "
Jul 28 18:14:37.118140 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.118116    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24780714-585d-4aa7-b287-9a981264b266-proxy-tls\") pod \"24780714-585d-4aa7-b287-9a981264b266\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") "
Jul 28 18:14:37.118482 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.118159    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24780714-585d-4aa7-b287-9a981264b266-kserve-provision-location\") pod \"24780714-585d-4aa7-b287-9a981264b266\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") "
Jul 28 18:14:37.118482 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.118188    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-cabundle-cert\") pod \"24780714-585d-4aa7-b287-9a981264b266\" (UID: \"24780714-585d-4aa7-b287-9a981264b266\") "
Jul 28 18:14:37.118595 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.118531    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24780714-585d-4aa7-b287-9a981264b266-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "24780714-585d-4aa7-b287-9a981264b266" (UID: "24780714-585d-4aa7-b287-9a981264b266"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:14:37.118646 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.118594    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "24780714-585d-4aa7-b287-9a981264b266" (UID: "24780714-585d-4aa7-b287-9a981264b266"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:14:37.118646 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.118608    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "24780714-585d-4aa7-b287-9a981264b266" (UID: "24780714-585d-4aa7-b287-9a981264b266"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:14:37.120267 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.120217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24780714-585d-4aa7-b287-9a981264b266-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24780714-585d-4aa7-b287-9a981264b266" (UID: "24780714-585d-4aa7-b287-9a981264b266"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:14:37.120364 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.120276    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24780714-585d-4aa7-b287-9a981264b266-kube-api-access-r875f" (OuterVolumeSpecName: "kube-api-access-r875f") pod "24780714-585d-4aa7-b287-9a981264b266" (UID: "24780714-585d-4aa7-b287-9a981264b266"). InnerVolumeSpecName "kube-api-access-r875f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:14:37.219601 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.219572    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:14:37.219601 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.219596    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r875f\" (UniqueName: \"kubernetes.io/projected/24780714-585d-4aa7-b287-9a981264b266-kube-api-access-r875f\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:14:37.219772 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.219607    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/24780714-585d-4aa7-b287-9a981264b266-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:14:37.219772 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.219616    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24780714-585d-4aa7-b287-9a981264b266-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:14:37.219772 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.219626    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/24780714-585d-4aa7-b287-9a981264b266-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:14:37.417438 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.417407    2577 generic.go:356] "Generic (PLEG): container finished" podID="24780714-585d-4aa7-b287-9a981264b266" containerID="5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506" exitCode=0
Jul 28 18:14:37.417597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.417445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" event={"ID":"24780714-585d-4aa7-b287-9a981264b266","Type":"ContainerDied","Data":"5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506"}
Jul 28 18:14:37.417597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.417464    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"
Jul 28 18:14:37.417597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.417494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr" event={"ID":"24780714-585d-4aa7-b287-9a981264b266","Type":"ContainerDied","Data":"788efa00e5af2a8cfa4fbd4d370133526a0dd2b4b2239088c87c2eb3d5268d42"}
Jul 28 18:14:37.417597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.417521    2577 scope.go:117] "RemoveContainer" containerID="18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293"
Jul 28 18:14:37.426298 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.426276    2577 scope.go:117] "RemoveContainer" containerID="5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506"
Jul 28 18:14:37.438544 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.438522    2577 scope.go:117] "RemoveContainer" containerID="1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97"
Jul 28 18:14:37.440995 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.440971    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"]
Jul 28 18:14:37.445137 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.445113    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-rksbr"]
Jul 28 18:14:37.446501 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.446486    2577 scope.go:117] "RemoveContainer" containerID="18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293"
Jul 28 18:14:37.446749 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:14:37.446730    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293\": container with ID starting with 18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293 not found: ID does not exist" containerID="18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293"
Jul 28 18:14:37.446794 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.446757    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293"} err="failed to get container status \"18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293\": rpc error: code = NotFound desc = could not find container \"18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293\": container with ID starting with 18b161b02b35f79c26a6555f14f9160f703555e375bf731374c2f89e2cb82293 not found: ID does not exist"
Jul 28 18:14:37.446794 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.446774    2577 scope.go:117] "RemoveContainer" containerID="5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506"
Jul 28 18:14:37.447025 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:14:37.447006    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506\": container with ID starting with 5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506 not found: ID does not exist" containerID="5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506"
Jul 28 18:14:37.447076 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.447031    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506"} err="failed to get container status \"5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506\": rpc error: code = NotFound desc = could not find container \"5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506\": container with ID starting with 5e93d5d1846e525ac42befc3ce00a3240db001a929abc7ae6060bcaa35a98506 not found: ID does not exist"
Jul 28 18:14:37.447076 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.447045    2577 scope.go:117] "RemoveContainer" containerID="1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97"
Jul 28 18:14:37.447329 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:14:37.447227    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97\": container with ID starting with 1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97 not found: ID does not exist" containerID="1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97"
Jul 28 18:14:37.447329 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.447262    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97"} err="failed to get container status \"1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97\": rpc error: code = NotFound desc = could not find container \"1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97\": container with ID starting with 1c10f08a5f4593bc116ce9f28e2537442d4732fb4c990a388b6206a13249cb97 not found: ID does not exist"
Jul 28 18:14:37.903975 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:37.903943    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24780714-585d-4aa7-b287-9a981264b266" path="/var/lib/kubelet/pods/24780714-585d-4aa7-b287-9a981264b266/volumes"
Jul 28 18:14:43.859149 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.859112    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"]
Jul 28 18:14:43.859658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.859459    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="storage-initializer"
Jul 28 18:14:43.859658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.859471    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="storage-initializer"
Jul 28 18:14:43.859658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.859482    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kube-rbac-proxy"
Jul 28 18:14:43.859658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.859488    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kube-rbac-proxy"
Jul 28 18:14:43.859658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.859499    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kserve-container"
Jul 28 18:14:43.859658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.859505    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kserve-container"
Jul 28 18:14:43.859658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.859563    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kserve-container"
Jul 28 18:14:43.859658 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.859574    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="24780714-585d-4aa7-b287-9a981264b266" containerName="kube-rbac-proxy"
Jul 28 18:14:43.864072 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.864051    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.867054 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.867033    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jul 28 18:14:43.867174 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.867076    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:14:43.867174 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.867154    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:14:43.867302 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.867159    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jul 28 18:14:43.868259 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.868227    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:14:43.868346 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.868302    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:14:43.868502 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.868484    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:14:43.871695 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.871671    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h256d\" (UniqueName: \"kubernetes.io/projected/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-api-access-h256d\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.871802 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.871741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-proxy-tls\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.871802 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.871773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.871802 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.871796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-cabundle-cert\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.871971 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.871852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kserve-provision-location\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.873555 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.873537    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"]
Jul 28 18:14:43.973062 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.973030    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-proxy-tls\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.973276 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.973072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.973276 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.973099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-cabundle-cert\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.973403 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.973269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kserve-provision-location\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.973403 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.973342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h256d\" (UniqueName: \"kubernetes.io/projected/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-api-access-h256d\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.973726 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.973696    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kserve-provision-location\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.973854 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.973793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.973904 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.973855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-cabundle-cert\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.975544 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.975518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-proxy-tls\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:43.981207 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:43.981185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h256d\" (UniqueName: \"kubernetes.io/projected/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-api-access-h256d\") pod \"isvc-sklearn-predictor-6765cc767b-k9hbh\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:44.175488 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:44.175463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:44.303318 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:44.303297    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"]
Jul 28 18:14:44.305293 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:14:44.305266    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcb117f_c01c_4c93_ac4d_ea13c566f9dc.slice/crio-271e99b088b1987ae6fed2a60495212750d567f68988dfa95b42415ba4684b22 WatchSource:0}: Error finding container 271e99b088b1987ae6fed2a60495212750d567f68988dfa95b42415ba4684b22: Status 404 returned error can't find the container with id 271e99b088b1987ae6fed2a60495212750d567f68988dfa95b42415ba4684b22
Jul 28 18:14:44.448788 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:44.448699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" event={"ID":"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc","Type":"ContainerStarted","Data":"b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da"}
Jul 28 18:14:44.448788 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:44.448771    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" event={"ID":"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc","Type":"ContainerStarted","Data":"271e99b088b1987ae6fed2a60495212750d567f68988dfa95b42415ba4684b22"}
Jul 28 18:14:48.472367 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:48.472336    2577 generic.go:356] "Generic (PLEG): container finished" podID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerID="b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da" exitCode=0
Jul 28 18:14:48.472749 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:48.472387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" event={"ID":"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc","Type":"ContainerDied","Data":"b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da"}
Jul 28 18:14:49.478599 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:49.478564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" event={"ID":"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc","Type":"ContainerStarted","Data":"b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e"}
Jul 28 18:14:49.478988 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:49.478632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" event={"ID":"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc","Type":"ContainerStarted","Data":"06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222"}
Jul 28 18:14:49.478988 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:49.478831    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:49.503261 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:49.503197    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podStartSLOduration=2.548781549 podStartE2EDuration="6.503184896s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.954403347s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:14:49.501707469 +0000 UTC m=+3848.145676960" watchObservedRunningTime="2026-07-28 18:14:49.503184896 +0000 UTC m=+3848.147154386"
Jul 28 18:14:50.482893 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:50.482865    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:50.484300 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:50.484268    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:14:51.487022 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:51.486983    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:14:56.491481 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:56.491450    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:14:56.491944 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:14:56.491915    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:15:06.492830 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:15:06.492746    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:15:16.492380 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:15:16.492336    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:15:26.492042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:15:26.492005    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:15:36.491900 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:15:36.491862    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:15:46.492452 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:15:46.492413    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:15:56.491919 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:15:56.491880    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:16:06.493158 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:06.493128    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:16:14.019447 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:14.019412    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"]
Jul 28 18:16:14.019837 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:14.019710    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" containerID="cri-o://06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222" gracePeriod=30
Jul 28 18:16:14.019837 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:14.019745    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kube-rbac-proxy" containerID="cri-o://b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e" gracePeriod=30
Jul 28 18:16:14.830582 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:14.830548    2577 generic.go:356] "Generic (PLEG): container finished" podID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerID="b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e" exitCode=2
Jul 28 18:16:14.830766 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:14.830619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" event={"ID":"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc","Type":"ContainerDied","Data":"b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e"}
Jul 28 18:16:16.047102 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:16.047077    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:16:16.054326 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:16.054303    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:16:16.487724 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:16.487679    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 28 18:16:16.491958 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:16.491928    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 18:16:18.563482 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.563460    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:16:18.598147 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.598089    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h256d\" (UniqueName: \"kubernetes.io/projected/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-api-access-h256d\") pod \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") "
Jul 28 18:16:18.598147 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.598125    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kserve-provision-location\") pod \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") "
Jul 28 18:16:18.598317 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.598180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-rbac-proxy-sar-config\") pod \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") "
Jul 28 18:16:18.598317 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.598217    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-proxy-tls\") pod \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") "
Jul 28 18:16:18.598317 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.598268    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-cabundle-cert\") pod \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\" (UID: \"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc\") "
Jul 28 18:16:18.598610 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.598526    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" (UID: "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:16:18.598704 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.598641    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" (UID: "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:16:18.598704 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.598670    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" (UID: "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:16:18.600702 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.600677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-api-access-h256d" (OuterVolumeSpecName: "kube-api-access-h256d") pod "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" (UID: "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc"). InnerVolumeSpecName "kube-api-access-h256d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:16:18.600817 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.600683    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" (UID: "9fcb117f-c01c-4c93-ac4d-ea13c566f9dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:16:18.698845 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.698825    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:16:18.698845 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.698845    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:16:18.698981 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.698854    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h256d\" (UniqueName: \"kubernetes.io/projected/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-api-access-h256d\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:16:18.698981 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.698865    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:16:18.698981 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.698874    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:16:18.850404 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.850326    2577 generic.go:356] "Generic (PLEG): container finished" podID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerID="06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222" exitCode=0
Jul 28 18:16:18.850404 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.850381    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"
Jul 28 18:16:18.850404 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.850391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" event={"ID":"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc","Type":"ContainerDied","Data":"06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222"}
Jul 28 18:16:18.850619 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.850439    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh" event={"ID":"9fcb117f-c01c-4c93-ac4d-ea13c566f9dc","Type":"ContainerDied","Data":"271e99b088b1987ae6fed2a60495212750d567f68988dfa95b42415ba4684b22"}
Jul 28 18:16:18.850619 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.850468    2577 scope.go:117] "RemoveContainer" containerID="b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e"
Jul 28 18:16:18.859453 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.859395    2577 scope.go:117] "RemoveContainer" containerID="06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222"
Jul 28 18:16:18.866378 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.866359    2577 scope.go:117] "RemoveContainer" containerID="b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da"
Jul 28 18:16:18.873184 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.873166    2577 scope.go:117] "RemoveContainer" containerID="b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e"
Jul 28 18:16:18.873508 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:16:18.873489    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e\": container with ID starting with b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e not found: ID does not exist" containerID="b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e"
Jul 28 18:16:18.873571 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.873515    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e"} err="failed to get container status \"b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e\": rpc error: code = NotFound desc = could not find container \"b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e\": container with ID starting with b56e7efb98ead991186be44fb5b4829fd00f4c10b07d3cdaec98c8eeace79d4e not found: ID does not exist"
Jul 28 18:16:18.873571 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.873531    2577 scope.go:117] "RemoveContainer" containerID="06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222"
Jul 28 18:16:18.873746 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:16:18.873724    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222\": container with ID starting with 06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222 not found: ID does not exist" containerID="06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222"
Jul 28 18:16:18.873819 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.873755    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222"} err="failed to get container status \"06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222\": rpc error: code = NotFound desc = could not find container \"06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222\": container with ID starting with 06fc30a3af18ebb4dece2aff4b3b5af22132324342851e8caeef034e201c1222 not found: ID does not exist"
Jul 28 18:16:18.873819 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.873777    2577 scope.go:117] "RemoveContainer" containerID="b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da"
Jul 28 18:16:18.873991 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:16:18.873976    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da\": container with ID starting with b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da not found: ID does not exist" containerID="b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da"
Jul 28 18:16:18.874026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.873994    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da"} err="failed to get container status \"b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da\": rpc error: code = NotFound desc = could not find container \"b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da\": container with ID starting with b2b229c5402f0a146e6a15c21e4e6cedaa3c898ae6856b621facced71fd940da not found: ID does not exist"
Jul 28 18:16:18.886388 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.886365    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"]
Jul 28 18:16:18.905456 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:18.905421    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-6765cc767b-k9hbh"]
Jul 28 18:16:19.235147 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.235100    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"]
Jul 28 18:16:19.235559 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.235541    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kube-rbac-proxy"
Jul 28 18:16:19.235653 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.235561    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kube-rbac-proxy"
Jul 28 18:16:19.235653 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.235574    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="storage-initializer"
Jul 28 18:16:19.235653 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.235582    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="storage-initializer"
Jul 28 18:16:19.235653 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.235609    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container"
Jul 28 18:16:19.235653 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.235618    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container"
Jul 28 18:16:19.235909 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.235685    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kube-rbac-proxy"
Jul 28 18:16:19.235909 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.235703    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" containerName="kserve-container"
Jul 28 18:16:19.240312 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.240291    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.243289 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.243265    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jul 28 18:16:19.243408 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.243330    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 18:16:19.243408 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.243334    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:16:19.243602 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.243586    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:16:19.243657 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.243646    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:16:19.243710 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.243667    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:16:19.243710 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.243651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:16:19.249189 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.249170    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"]
Jul 28 18:16:19.302660 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.302623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da845ee-8025-4b50-b251-4dee2ce13566-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.302660 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.302664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.302921 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.302701    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.302921 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.302815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da845ee-8025-4b50-b251-4dee2ce13566-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.302921 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.302905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-869n2\" (UniqueName: \"kubernetes.io/projected/0da845ee-8025-4b50-b251-4dee2ce13566-kube-api-access-869n2\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.403312 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.403277    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da845ee-8025-4b50-b251-4dee2ce13566-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.403470 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.403325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.403470 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.403361    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.403576 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.403541    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da845ee-8025-4b50-b251-4dee2ce13566-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.403669 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.403650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-869n2\" (UniqueName: \"kubernetes.io/projected/0da845ee-8025-4b50-b251-4dee2ce13566-kube-api-access-869n2\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.403714 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:16:19.403671    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Jul 28 18:16:19.403766 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:16:19.403756    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0da845ee-8025-4b50-b251-4dee2ce13566-proxy-tls podName:0da845ee-8025-4b50-b251-4dee2ce13566 nodeName:}" failed. No retries permitted until 2026-07-28 18:16:19.903733856 +0000 UTC m=+3938.547703341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0da845ee-8025-4b50-b251-4dee2ce13566-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" (UID: "0da845ee-8025-4b50-b251-4dee2ce13566") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Jul 28 18:16:19.403830 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.403759    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da845ee-8025-4b50-b251-4dee2ce13566-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.404013 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.403996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.404106 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.404088    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.413100 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.413076    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-869n2\" (UniqueName: \"kubernetes.io/projected/0da845ee-8025-4b50-b251-4dee2ce13566-kube-api-access-869n2\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.903812 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.903774    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fcb117f-c01c-4c93-ac4d-ea13c566f9dc" path="/var/lib/kubelet/pods/9fcb117f-c01c-4c93-ac4d-ea13c566f9dc/volumes"
Jul 28 18:16:19.908067 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.908046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da845ee-8025-4b50-b251-4dee2ce13566-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:19.910370 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:19.910351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da845ee-8025-4b50-b251-4dee2ce13566-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:20.152624 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:20.152589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:20.292617 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:20.292594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"]
Jul 28 18:16:20.295181 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:16:20.295156    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da845ee_8025_4b50_b251_4dee2ce13566.slice/crio-ced824fae4fcf19e108f23e96465524f806ea6c5231cce462157eb591e812899 WatchSource:0}: Error finding container ced824fae4fcf19e108f23e96465524f806ea6c5231cce462157eb591e812899: Status 404 returned error can't find the container with id ced824fae4fcf19e108f23e96465524f806ea6c5231cce462157eb591e812899
Jul 28 18:16:20.860352 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:20.860317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" event={"ID":"0da845ee-8025-4b50-b251-4dee2ce13566","Type":"ContainerStarted","Data":"cb04a67afc92be4d0089d5ffcf7f5fed9c53c388a560a1a7fed90d44310e0665"}
Jul 28 18:16:20.860558 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:20.860385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" event={"ID":"0da845ee-8025-4b50-b251-4dee2ce13566","Type":"ContainerStarted","Data":"ced824fae4fcf19e108f23e96465524f806ea6c5231cce462157eb591e812899"}
Jul 28 18:16:24.878239 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:24.878200    2577 generic.go:356] "Generic (PLEG): container finished" podID="0da845ee-8025-4b50-b251-4dee2ce13566" containerID="cb04a67afc92be4d0089d5ffcf7f5fed9c53c388a560a1a7fed90d44310e0665" exitCode=0
Jul 28 18:16:24.878700 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:24.878274    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" event={"ID":"0da845ee-8025-4b50-b251-4dee2ce13566","Type":"ContainerDied","Data":"cb04a67afc92be4d0089d5ffcf7f5fed9c53c388a560a1a7fed90d44310e0665"}
Jul 28 18:16:25.884836 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:25.884802    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" event={"ID":"0da845ee-8025-4b50-b251-4dee2ce13566","Type":"ContainerStarted","Data":"66162c956ed3d55bfde942eaebeb7fdc2937e363409eeb2c34ed96e7f7dbcdba"}
Jul 28 18:16:25.885189 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:25.884862    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" event={"ID":"0da845ee-8025-4b50-b251-4dee2ce13566","Type":"ContainerStarted","Data":"ad48b92cde03653587ff135ca3dc276cf7e2c1aaa0e8f4b2dd00faf5a6581264"}
Jul 28 18:16:25.885189 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:25.885160    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:25.885322 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:25.885206    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:16:25.918787 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:25.918742    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" podStartSLOduration=3.239856233 podStartE2EDuration="6.918720619s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.678864386s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:16:25.916620082 +0000 UTC m=+3944.560589572" watchObservedRunningTime="2026-07-28 18:16:25.918720619 +0000 UTC m=+3944.562690109"
Jul 28 18:16:31.893549 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:16:31.893520    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:17:01.933619 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:01.933568    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 18:17:11.896227 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:11.896196    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:17:19.287344 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:19.287312    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"]
Jul 28 18:17:19.287762 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:19.287525    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kserve-container" containerID="cri-o://ad48b92cde03653587ff135ca3dc276cf7e2c1aaa0e8f4b2dd00faf5a6581264" gracePeriod=30
Jul 28 18:17:19.287762 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:19.287592    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kube-rbac-proxy" containerID="cri-o://66162c956ed3d55bfde942eaebeb7fdc2937e363409eeb2c34ed96e7f7dbcdba" gracePeriod=30
Jul 28 18:17:20.093562 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:20.093524    2577 generic.go:356] "Generic (PLEG): container finished" podID="0da845ee-8025-4b50-b251-4dee2ce13566" containerID="66162c956ed3d55bfde942eaebeb7fdc2937e363409eeb2c34ed96e7f7dbcdba" exitCode=2
Jul 28 18:17:20.093702 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:20.093579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" event={"ID":"0da845ee-8025-4b50-b251-4dee2ce13566","Type":"ContainerDied","Data":"66162c956ed3d55bfde942eaebeb7fdc2937e363409eeb2c34ed96e7f7dbcdba"}
Jul 28 18:17:21.889474 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:21.889432    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 28 18:17:24.526702 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.526671    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"]
Jul 28 18:17:24.530623 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.530600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.533408 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.533383    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jul 28 18:17:24.533518 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.533447    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:17:24.542632 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.542607    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"]
Jul 28 18:17:24.607394 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.607373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18751f74-e23f-406f-8795-e8fc71279ea6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.607505 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.607420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18751f74-e23f-406f-8795-e8fc71279ea6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.607505 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.607443    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rrf\" (UniqueName: \"kubernetes.io/projected/18751f74-e23f-406f-8795-e8fc71279ea6-kube-api-access-m7rrf\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.607505 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.607499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.607616 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.607542    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.707887 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.707859    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18751f74-e23f-406f-8795-e8fc71279ea6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.708034 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.707911    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18751f74-e23f-406f-8795-e8fc71279ea6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.708087 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.708026    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rrf\" (UniqueName: \"kubernetes.io/projected/18751f74-e23f-406f-8795-e8fc71279ea6-kube-api-access-m7rrf\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.708087 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.708064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.708189 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.708107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.708239 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.708221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18751f74-e23f-406f-8795-e8fc71279ea6-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.708692 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.708673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.708755 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.708703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.710412 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.710386    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18751f74-e23f-406f-8795-e8fc71279ea6-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.716848 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.716829    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rrf\" (UniqueName: \"kubernetes.io/projected/18751f74-e23f-406f-8795-e8fc71279ea6-kube-api-access-m7rrf\") pod \"isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.840991 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.840904    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:24.972974 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:24.972953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"]
Jul 28 18:17:24.974671 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:17:24.974638    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18751f74_e23f_406f_8795_e8fc71279ea6.slice/crio-6299718ff563a6d4279b5c629a4abb84a120f3d4138322ebec76a116659c14d5 WatchSource:0}: Error finding container 6299718ff563a6d4279b5c629a4abb84a120f3d4138322ebec76a116659c14d5: Status 404 returned error can't find the container with id 6299718ff563a6d4279b5c629a4abb84a120f3d4138322ebec76a116659c14d5
Jul 28 18:17:25.115601 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:25.115521    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" event={"ID":"18751f74-e23f-406f-8795-e8fc71279ea6","Type":"ContainerStarted","Data":"95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623"}
Jul 28 18:17:25.115601 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:25.115577    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" event={"ID":"18751f74-e23f-406f-8795-e8fc71279ea6","Type":"ContainerStarted","Data":"6299718ff563a6d4279b5c629a4abb84a120f3d4138322ebec76a116659c14d5"}
Jul 28 18:17:26.123395 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.123358    2577 generic.go:356] "Generic (PLEG): container finished" podID="0da845ee-8025-4b50-b251-4dee2ce13566" containerID="ad48b92cde03653587ff135ca3dc276cf7e2c1aaa0e8f4b2dd00faf5a6581264" exitCode=0
Jul 28 18:17:26.123734 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.123431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" event={"ID":"0da845ee-8025-4b50-b251-4dee2ce13566","Type":"ContainerDied","Data":"ad48b92cde03653587ff135ca3dc276cf7e2c1aaa0e8f4b2dd00faf5a6581264"}
Jul 28 18:17:26.234888 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.234864    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:17:26.322016 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.321912    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-869n2\" (UniqueName: \"kubernetes.io/projected/0da845ee-8025-4b50-b251-4dee2ce13566-kube-api-access-869n2\") pod \"0da845ee-8025-4b50-b251-4dee2ce13566\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") "
Jul 28 18:17:26.322016 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.321956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-kube-rbac-proxy-sar-config\") pod \"0da845ee-8025-4b50-b251-4dee2ce13566\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") "
Jul 28 18:17:26.322016 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.321988    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da845ee-8025-4b50-b251-4dee2ce13566-proxy-tls\") pod \"0da845ee-8025-4b50-b251-4dee2ce13566\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") "
Jul 28 18:17:26.322016 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.322018    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da845ee-8025-4b50-b251-4dee2ce13566-kserve-provision-location\") pod \"0da845ee-8025-4b50-b251-4dee2ce13566\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") "
Jul 28 18:17:26.322379 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.322049    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-cabundle-cert\") pod \"0da845ee-8025-4b50-b251-4dee2ce13566\" (UID: \"0da845ee-8025-4b50-b251-4dee2ce13566\") "
Jul 28 18:17:26.322426 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.322377    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da845ee-8025-4b50-b251-4dee2ce13566-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0da845ee-8025-4b50-b251-4dee2ce13566" (UID: "0da845ee-8025-4b50-b251-4dee2ce13566"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:17:26.322426 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.322331    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0da845ee-8025-4b50-b251-4dee2ce13566" (UID: "0da845ee-8025-4b50-b251-4dee2ce13566"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:17:26.322511 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.322485    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0da845ee-8025-4b50-b251-4dee2ce13566" (UID: "0da845ee-8025-4b50-b251-4dee2ce13566"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:17:26.324148 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.324127    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0da845ee-8025-4b50-b251-4dee2ce13566-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0da845ee-8025-4b50-b251-4dee2ce13566" (UID: "0da845ee-8025-4b50-b251-4dee2ce13566"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:17:26.324225 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.324144    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da845ee-8025-4b50-b251-4dee2ce13566-kube-api-access-869n2" (OuterVolumeSpecName: "kube-api-access-869n2") pod "0da845ee-8025-4b50-b251-4dee2ce13566" (UID: "0da845ee-8025-4b50-b251-4dee2ce13566"). InnerVolumeSpecName "kube-api-access-869n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:17:26.422850 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.422812    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-869n2\" (UniqueName: \"kubernetes.io/projected/0da845ee-8025-4b50-b251-4dee2ce13566-kube-api-access-869n2\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:26.422850 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.422843    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:26.422850 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.422853    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0da845ee-8025-4b50-b251-4dee2ce13566-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:26.423073 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.422862    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0da845ee-8025-4b50-b251-4dee2ce13566-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:26.423073 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:26.422871    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0da845ee-8025-4b50-b251-4dee2ce13566-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:27.129689 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:27.129653    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2" event={"ID":"0da845ee-8025-4b50-b251-4dee2ce13566","Type":"ContainerDied","Data":"ced824fae4fcf19e108f23e96465524f806ea6c5231cce462157eb591e812899"}
Jul 28 18:17:27.129689 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:27.129671    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"
Jul 28 18:17:27.130177 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:27.129721    2577 scope.go:117] "RemoveContainer" containerID="66162c956ed3d55bfde942eaebeb7fdc2937e363409eeb2c34ed96e7f7dbcdba"
Jul 28 18:17:27.138092 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:27.138074    2577 scope.go:117] "RemoveContainer" containerID="ad48b92cde03653587ff135ca3dc276cf7e2c1aaa0e8f4b2dd00faf5a6581264"
Jul 28 18:17:27.145217 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:27.145202    2577 scope.go:117] "RemoveContainer" containerID="cb04a67afc92be4d0089d5ffcf7f5fed9c53c388a560a1a7fed90d44310e0665"
Jul 28 18:17:27.153305 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:27.153290    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"]
Jul 28 18:17:27.155943 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:27.155920    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-qd2l2"]
Jul 28 18:17:27.904335 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:27.904301    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" path="/var/lib/kubelet/pods/0da845ee-8025-4b50-b251-4dee2ce13566/volumes"
Jul 28 18:17:31.148703 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:31.148673    2577 generic.go:356] "Generic (PLEG): container finished" podID="18751f74-e23f-406f-8795-e8fc71279ea6" containerID="95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623" exitCode=0
Jul 28 18:17:31.149026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:31.148730    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" event={"ID":"18751f74-e23f-406f-8795-e8fc71279ea6","Type":"ContainerDied","Data":"95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623"}
Jul 28 18:17:32.154993 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:32.154955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" event={"ID":"18751f74-e23f-406f-8795-e8fc71279ea6","Type":"ContainerStarted","Data":"ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a"}
Jul 28 18:17:32.155431 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:32.155020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" event={"ID":"18751f74-e23f-406f-8795-e8fc71279ea6","Type":"ContainerStarted","Data":"ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8"}
Jul 28 18:17:32.155431 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:32.155311    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:32.155547 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:32.155482    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:32.156745 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:32.156715    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 18:17:32.178347 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:32.178291    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" podStartSLOduration=2.178153309 podStartE2EDuration="8.178274823s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.000121514s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:17:32.176481349 +0000 UTC m=+4010.820450838" watchObservedRunningTime="2026-07-28 18:17:32.178274823 +0000 UTC m=+4010.822244315"
Jul 28 18:17:33.159055 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:33.159014    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 18:17:38.163127 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:38.163099    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:38.163733 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:38.163701    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 18:17:48.164934 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:48.164901    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:56.405571 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:56.405539    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv_18751f74-e23f-406f-8795-e8fc71279ea6/kserve-container/0.log"
Jul 28 18:17:56.592508 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:56.592474    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"]
Jul 28 18:17:56.592786 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:56.592746    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kserve-container" containerID="cri-o://ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8" gracePeriod=30
Jul 28 18:17:56.592931 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:56.592910    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kube-rbac-proxy" containerID="cri-o://ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a" gracePeriod=30
Jul 28 18:17:57.267939 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.267905    2577 generic.go:356] "Generic (PLEG): container finished" podID="18751f74-e23f-406f-8795-e8fc71279ea6" containerID="ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a" exitCode=2
Jul 28 18:17:57.268109 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.267972    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" event={"ID":"18751f74-e23f-406f-8795-e8fc71279ea6","Type":"ContainerDied","Data":"ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a"}
Jul 28 18:17:57.733854 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.733832    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:57.782040 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.782014    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-kube-rbac-proxy-sar-config\") pod \"18751f74-e23f-406f-8795-e8fc71279ea6\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") "
Jul 28 18:17:57.782196 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.782058    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18751f74-e23f-406f-8795-e8fc71279ea6-proxy-tls\") pod \"18751f74-e23f-406f-8795-e8fc71279ea6\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") "
Jul 28 18:17:57.782196 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.782080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18751f74-e23f-406f-8795-e8fc71279ea6-kserve-provision-location\") pod \"18751f74-e23f-406f-8795-e8fc71279ea6\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") "
Jul 28 18:17:57.782196 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.782099    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-cabundle-cert\") pod \"18751f74-e23f-406f-8795-e8fc71279ea6\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") "
Jul 28 18:17:57.782396 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.782275    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7rrf\" (UniqueName: \"kubernetes.io/projected/18751f74-e23f-406f-8795-e8fc71279ea6-kube-api-access-m7rrf\") pod \"18751f74-e23f-406f-8795-e8fc71279ea6\" (UID: \"18751f74-e23f-406f-8795-e8fc71279ea6\") "
Jul 28 18:17:57.782454 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.782438    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "18751f74-e23f-406f-8795-e8fc71279ea6" (UID: "18751f74-e23f-406f-8795-e8fc71279ea6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:17:57.782512 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.782449    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "18751f74-e23f-406f-8795-e8fc71279ea6" (UID: "18751f74-e23f-406f-8795-e8fc71279ea6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:17:57.782619 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.782601    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:57.782668 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.782624    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/18751f74-e23f-406f-8795-e8fc71279ea6-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:57.784615 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.784589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18751f74-e23f-406f-8795-e8fc71279ea6-kube-api-access-m7rrf" (OuterVolumeSpecName: "kube-api-access-m7rrf") pod "18751f74-e23f-406f-8795-e8fc71279ea6" (UID: "18751f74-e23f-406f-8795-e8fc71279ea6"). InnerVolumeSpecName "kube-api-access-m7rrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:17:57.784720 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.784642    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18751f74-e23f-406f-8795-e8fc71279ea6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "18751f74-e23f-406f-8795-e8fc71279ea6" (UID: "18751f74-e23f-406f-8795-e8fc71279ea6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:17:57.784720 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.784651    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18751f74-e23f-406f-8795-e8fc71279ea6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "18751f74-e23f-406f-8795-e8fc71279ea6" (UID: "18751f74-e23f-406f-8795-e8fc71279ea6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:17:57.883884 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.883819    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7rrf\" (UniqueName: \"kubernetes.io/projected/18751f74-e23f-406f-8795-e8fc71279ea6-kube-api-access-m7rrf\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:57.883884 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.883842    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/18751f74-e23f-406f-8795-e8fc71279ea6-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:57.883884 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:57.883852    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/18751f74-e23f-406f-8795-e8fc71279ea6-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:17:58.276284 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.276238    2577 generic.go:356] "Generic (PLEG): container finished" podID="18751f74-e23f-406f-8795-e8fc71279ea6" containerID="ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8" exitCode=0
Jul 28 18:17:58.276458 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.276281    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" event={"ID":"18751f74-e23f-406f-8795-e8fc71279ea6","Type":"ContainerDied","Data":"ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8"}
Jul 28 18:17:58.276458 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.276307    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"
Jul 28 18:17:58.276458 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.276366    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv" event={"ID":"18751f74-e23f-406f-8795-e8fc71279ea6","Type":"ContainerDied","Data":"6299718ff563a6d4279b5c629a4abb84a120f3d4138322ebec76a116659c14d5"}
Jul 28 18:17:58.276458 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.276407    2577 scope.go:117] "RemoveContainer" containerID="ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a"
Jul 28 18:17:58.285987 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.285970    2577 scope.go:117] "RemoveContainer" containerID="ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8"
Jul 28 18:17:58.295123 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.295047    2577 scope.go:117] "RemoveContainer" containerID="95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623"
Jul 28 18:17:58.303524 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.303484    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"]
Jul 28 18:17:58.303866 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.303844    2577 scope.go:117] "RemoveContainer" containerID="ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a"
Jul 28 18:17:58.304142 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:17:58.304116    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a\": container with ID starting with ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a not found: ID does not exist" containerID="ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a"
Jul 28 18:17:58.304296 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.304150    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a"} err="failed to get container status \"ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a\": rpc error: code = NotFound desc = could not find container \"ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a\": container with ID starting with ee2308e245908df9689380419dd2ce57c8cd2b9da16da183652c1b9fb785763a not found: ID does not exist"
Jul 28 18:17:58.304296 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.304174    2577 scope.go:117] "RemoveContainer" containerID="ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8"
Jul 28 18:17:58.304464 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:17:58.304438    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8\": container with ID starting with ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8 not found: ID does not exist" containerID="ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8"
Jul 28 18:17:58.304528 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.304474    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8"} err="failed to get container status \"ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8\": rpc error: code = NotFound desc = could not find container \"ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8\": container with ID starting with ef19419b1b5a3e116cf4d9ff7a6ea498f1d14c4f78ecc7515e89441fbc1f07b8 not found: ID does not exist"
Jul 28 18:17:58.304528 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.304495    2577 scope.go:117] "RemoveContainer" containerID="95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623"
Jul 28 18:17:58.304773 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:17:58.304751    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623\": container with ID starting with 95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623 not found: ID does not exist" containerID="95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623"
Jul 28 18:17:58.304842 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.304779    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623"} err="failed to get container status \"95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623\": rpc error: code = NotFound desc = could not find container \"95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623\": container with ID starting with 95641797ed35de31aee51cd97bd09b395135d4a690e2f21820c770555cedc623 not found: ID does not exist"
Jul 28 18:17:58.307896 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:58.307876    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-78ff5f86d8-kp7dv"]
Jul 28 18:17:59.904861 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:17:59.904827    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" path="/var/lib/kubelet/pods/18751f74-e23f-406f-8795-e8fc71279ea6/volumes"
Jul 28 18:18:01.741478 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741444    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"]
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741753    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="storage-initializer"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741763    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="storage-initializer"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741782    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kserve-container"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741787    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kserve-container"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741794    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="storage-initializer"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741800    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="storage-initializer"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741808    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kserve-container"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741814    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kserve-container"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741822    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kube-rbac-proxy"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741826    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kube-rbac-proxy"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741834    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kube-rbac-proxy"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741839    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kube-rbac-proxy"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741883    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kserve-container"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741891    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kserve-container"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741900    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da845ee-8025-4b50-b251-4dee2ce13566" containerName="kube-rbac-proxy"
Jul 28 18:18:01.742042 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.741907    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="18751f74-e23f-406f-8795-e8fc71279ea6" containerName="kube-rbac-proxy"
Jul 28 18:18:01.743887 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.743871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.746632 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.746614    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jul 28 18:18:01.746826 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.746659    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:18:01.747002 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.746868    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:18:01.747002 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.746891    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:18:01.747002 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.746909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:18:01.747166 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.747152    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:18:01.748291 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.748272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:18:01.755880 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.755861    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"]
Jul 28 18:18:01.812379 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.812357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e7c7c85-2904-44d9-9f65-a1f324a23364-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.812514 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.812396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.812514 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.812423    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265bj\" (UniqueName: \"kubernetes.io/projected/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-api-access-265bj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.812593 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.812537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.812593 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.812564    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e7c7c85-2904-44d9-9f65-a1f324a23364-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.913557 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.913532    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.913557 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.913558    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e7c7c85-2904-44d9-9f65-a1f324a23364-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.913750 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.913581    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e7c7c85-2904-44d9-9f65-a1f324a23364-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.913750 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.913612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.913750 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.913699    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265bj\" (UniqueName: \"kubernetes.io/projected/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-api-access-265bj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.914087 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.914063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e7c7c85-2904-44d9-9f65-a1f324a23364-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.914167 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.914147    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.914268 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.914233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.915991 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.915972    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e7c7c85-2904-44d9-9f65-a1f324a23364-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:01.921939 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:01.921917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-265bj\" (UniqueName: \"kubernetes.io/projected/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-api-access-265bj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:02.054119 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:02.054062    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:02.204120 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:02.204084    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"]
Jul 28 18:18:02.208278 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:18:02.208238    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7c7c85_2904_44d9_9f65_a1f324a23364.slice/crio-4ab486d5b2166cebf0b1d08a4d664bae758e4c1e4f47f7b2e4d89ee4ec1292f8 WatchSource:0}: Error finding container 4ab486d5b2166cebf0b1d08a4d664bae758e4c1e4f47f7b2e4d89ee4ec1292f8: Status 404 returned error can't find the container with id 4ab486d5b2166cebf0b1d08a4d664bae758e4c1e4f47f7b2e4d89ee4ec1292f8
Jul 28 18:18:02.295777 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:02.295744    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" event={"ID":"4e7c7c85-2904-44d9-9f65-a1f324a23364","Type":"ContainerStarted","Data":"917a3afc010b90db7fcc3e13283b70056103df94c461b31232453e209f6e6a04"}
Jul 28 18:18:02.295921 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:02.295800    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" event={"ID":"4e7c7c85-2904-44d9-9f65-a1f324a23364","Type":"ContainerStarted","Data":"4ab486d5b2166cebf0b1d08a4d664bae758e4c1e4f47f7b2e4d89ee4ec1292f8"}
Jul 28 18:18:06.312644 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:06.312613    2577 generic.go:356] "Generic (PLEG): container finished" podID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerID="917a3afc010b90db7fcc3e13283b70056103df94c461b31232453e209f6e6a04" exitCode=0
Jul 28 18:18:06.313000 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:06.312685    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" event={"ID":"4e7c7c85-2904-44d9-9f65-a1f324a23364","Type":"ContainerDied","Data":"917a3afc010b90db7fcc3e13283b70056103df94c461b31232453e209f6e6a04"}
Jul 28 18:18:07.318395 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:07.318363    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" event={"ID":"4e7c7c85-2904-44d9-9f65-a1f324a23364","Type":"ContainerStarted","Data":"9142fb1d639b31008371df0f9d19768a2bde295ad2ccf4bad2b1e59f3d91dcd4"}
Jul 28 18:18:07.318788 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:07.318425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" event={"ID":"4e7c7c85-2904-44d9-9f65-a1f324a23364","Type":"ContainerStarted","Data":"d101b04ba0af4c55a44267f5addedb720b982894c2f11c6dd8a7b425bd295b27"}
Jul 28 18:18:07.318788 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:07.318652    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:07.342458 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:07.342412    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" podStartSLOduration=2.3515498790000002 podStartE2EDuration="6.342396465s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.990846586s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:18:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:18:07.340215892 +0000 UTC m=+4045.984185381" watchObservedRunningTime="2026-07-28 18:18:07.342396465 +0000 UTC m=+4045.986365953"
Jul 28 18:18:08.322148 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:08.322125    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:14.331610 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:14.331579    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:18:44.335800 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:44.335757    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 18:18:54.334564 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:18:54.334534    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:19:01.845421 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:01.845385    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"]
Jul 28 18:19:01.845812 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:01.845668    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kserve-container" containerID="cri-o://d101b04ba0af4c55a44267f5addedb720b982894c2f11c6dd8a7b425bd295b27" gracePeriod=30
Jul 28 18:19:01.845812 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:01.845750    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kube-rbac-proxy" containerID="cri-o://9142fb1d639b31008371df0f9d19768a2bde295ad2ccf4bad2b1e59f3d91dcd4" gracePeriod=30
Jul 28 18:19:02.549819 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:02.549789    2577 generic.go:356] "Generic (PLEG): container finished" podID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerID="9142fb1d639b31008371df0f9d19768a2bde295ad2ccf4bad2b1e59f3d91dcd4" exitCode=2
Jul 28 18:19:02.549988 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:02.549850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" event={"ID":"4e7c7c85-2904-44d9-9f65-a1f324a23364","Type":"ContainerDied","Data":"9142fb1d639b31008371df0f9d19768a2bde295ad2ccf4bad2b1e59f3d91dcd4"}
Jul 28 18:19:04.327759 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:04.327720    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 28 18:19:08.578980 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.578948    2577 generic.go:356] "Generic (PLEG): container finished" podID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerID="d101b04ba0af4c55a44267f5addedb720b982894c2f11c6dd8a7b425bd295b27" exitCode=0
Jul 28 18:19:08.579420 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.579019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" event={"ID":"4e7c7c85-2904-44d9-9f65-a1f324a23364","Type":"ContainerDied","Data":"d101b04ba0af4c55a44267f5addedb720b982894c2f11c6dd8a7b425bd295b27"}
Jul 28 18:19:08.690588 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.690563    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:19:08.728634 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.728604    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e7c7c85-2904-44d9-9f65-a1f324a23364-proxy-tls\") pod \"4e7c7c85-2904-44d9-9f65-a1f324a23364\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") "
Jul 28 18:19:08.728798 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.728654    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-rbac-proxy-sar-config\") pod \"4e7c7c85-2904-44d9-9f65-a1f324a23364\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") "
Jul 28 18:19:08.728798 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.728703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e7c7c85-2904-44d9-9f65-a1f324a23364-kserve-provision-location\") pod \"4e7c7c85-2904-44d9-9f65-a1f324a23364\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") "
Jul 28 18:19:08.728798 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.728746    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-cabundle-cert\") pod \"4e7c7c85-2904-44d9-9f65-a1f324a23364\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") "
Jul 28 18:19:08.728798 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.728780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-265bj\" (UniqueName: \"kubernetes.io/projected/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-api-access-265bj\") pod \"4e7c7c85-2904-44d9-9f65-a1f324a23364\" (UID: \"4e7c7c85-2904-44d9-9f65-a1f324a23364\") "
Jul 28 18:19:08.729094 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.729069    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7c7c85-2904-44d9-9f65-a1f324a23364-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4e7c7c85-2904-44d9-9f65-a1f324a23364" (UID: "4e7c7c85-2904-44d9-9f65-a1f324a23364"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:19:08.729180 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.729091    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4e7c7c85-2904-44d9-9f65-a1f324a23364" (UID: "4e7c7c85-2904-44d9-9f65-a1f324a23364"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:19:08.729180 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.729112    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4e7c7c85-2904-44d9-9f65-a1f324a23364" (UID: "4e7c7c85-2904-44d9-9f65-a1f324a23364"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:19:08.730753 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.730724    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e7c7c85-2904-44d9-9f65-a1f324a23364-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4e7c7c85-2904-44d9-9f65-a1f324a23364" (UID: "4e7c7c85-2904-44d9-9f65-a1f324a23364"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:19:08.730882 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.730820    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-api-access-265bj" (OuterVolumeSpecName: "kube-api-access-265bj") pod "4e7c7c85-2904-44d9-9f65-a1f324a23364" (UID: "4e7c7c85-2904-44d9-9f65-a1f324a23364"). InnerVolumeSpecName "kube-api-access-265bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:19:08.829828 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.829752    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:19:08.829828 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.829779    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-265bj\" (UniqueName: \"kubernetes.io/projected/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-api-access-265bj\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:19:08.829828 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.829788    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4e7c7c85-2904-44d9-9f65-a1f324a23364-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:19:08.829828 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.829796    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4e7c7c85-2904-44d9-9f65-a1f324a23364-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:19:08.829828 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:08.829804    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4e7c7c85-2904-44d9-9f65-a1f324a23364-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:19:09.584841 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:09.584803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp" event={"ID":"4e7c7c85-2904-44d9-9f65-a1f324a23364","Type":"ContainerDied","Data":"4ab486d5b2166cebf0b1d08a4d664bae758e4c1e4f47f7b2e4d89ee4ec1292f8"}
Jul 28 18:19:09.585263 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:09.584847    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"
Jul 28 18:19:09.585263 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:09.584888    2577 scope.go:117] "RemoveContainer" containerID="9142fb1d639b31008371df0f9d19768a2bde295ad2ccf4bad2b1e59f3d91dcd4"
Jul 28 18:19:09.593233 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:09.593212    2577 scope.go:117] "RemoveContainer" containerID="d101b04ba0af4c55a44267f5addedb720b982894c2f11c6dd8a7b425bd295b27"
Jul 28 18:19:09.600652 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:09.600635    2577 scope.go:117] "RemoveContainer" containerID="917a3afc010b90db7fcc3e13283b70056103df94c461b31232453e209f6e6a04"
Jul 28 18:19:09.608405 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:09.608387    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"]
Jul 28 18:19:09.612331 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:09.612311    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-s5ggp"]
Jul 28 18:19:09.904296 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:19:09.904267    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" path="/var/lib/kubelet/pods/4e7c7c85-2904-44d9-9f65-a1f324a23364/volumes"
Jul 28 18:20:32.379546 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.379506    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"]
Jul 28 18:20:32.380060 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.379959    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kserve-container"
Jul 28 18:20:32.380060 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.379978    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kserve-container"
Jul 28 18:20:32.380060 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.380002    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="storage-initializer"
Jul 28 18:20:32.380060 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.380011    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="storage-initializer"
Jul 28 18:20:32.380060 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.380029    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kube-rbac-proxy"
Jul 28 18:20:32.380060 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.380038    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kube-rbac-proxy"
Jul 28 18:20:32.380426 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.380124    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kserve-container"
Jul 28 18:20:32.380426 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.380139    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e7c7c85-2904-44d9-9f65-a1f324a23364" containerName="kube-rbac-proxy"
Jul 28 18:20:32.383225 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.383204    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.386178 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.386156    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:20:32.386335 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.386157    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jul 28 18:20:32.386503 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.386172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:20:32.386503 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.386196    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jul 28 18:20:32.386503 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.386203    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:20:32.387510 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.387492    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:20:32.387608 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.387527    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:20:32.394444 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.394422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"]
Jul 28 18:20:32.481631 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.481603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqqrr\" (UniqueName: \"kubernetes.io/projected/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-api-access-zqqrr\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.481782 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.481644    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.481782 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.481660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.481855 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.481774    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.481855 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.481814    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.582951 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.582719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.583110 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.582968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.583110 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.583016    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqqrr\" (UniqueName: \"kubernetes.io/projected/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-api-access-zqqrr\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.583110 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.583063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.583110 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.583081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.583385 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.583362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.583462 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.583443    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.583707 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.583682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.585507 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.585489    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.592436 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.592410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqqrr\" (UniqueName: \"kubernetes.io/projected/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-api-access-zqqrr\") pod \"isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.694671 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.694632    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:32.821846 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.821813    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"]
Jul 28 18:20:32.824195 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:20:32.824164    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef87f0ea_95b0_4180_a0e5_6f8144d4711a.slice/crio-5d0ebb7ccd420f51f4b5a9267bb3af83e5dca1462683d99480d5b0840dfc0e22 WatchSource:0}: Error finding container 5d0ebb7ccd420f51f4b5a9267bb3af83e5dca1462683d99480d5b0840dfc0e22: Status 404 returned error can't find the container with id 5d0ebb7ccd420f51f4b5a9267bb3af83e5dca1462683d99480d5b0840dfc0e22
Jul 28 18:20:32.908154 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.908115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" event={"ID":"ef87f0ea-95b0-4180-a0e5-6f8144d4711a","Type":"ContainerStarted","Data":"2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398"}
Jul 28 18:20:32.908318 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:32.908176    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" event={"ID":"ef87f0ea-95b0-4180-a0e5-6f8144d4711a","Type":"ContainerStarted","Data":"5d0ebb7ccd420f51f4b5a9267bb3af83e5dca1462683d99480d5b0840dfc0e22"}
Jul 28 18:20:36.924825 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:36.924789    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerID="2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398" exitCode=0
Jul 28 18:20:36.925208 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:36.924861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" event={"ID":"ef87f0ea-95b0-4180-a0e5-6f8144d4711a","Type":"ContainerDied","Data":"2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398"}
Jul 28 18:20:37.930986 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:37.930952    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" event={"ID":"ef87f0ea-95b0-4180-a0e5-6f8144d4711a","Type":"ContainerStarted","Data":"1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a"}
Jul 28 18:20:37.931382 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:37.931006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" event={"ID":"ef87f0ea-95b0-4180-a0e5-6f8144d4711a","Type":"ContainerStarted","Data":"2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a"}
Jul 28 18:20:37.931382 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:37.931276    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:37.952795 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:37.952746    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podStartSLOduration=2.047687139 podStartE2EDuration="5.952731447s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.905044308s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:20:37.952498833 +0000 UTC m=+4196.596468324" watchObservedRunningTime="2026-07-28 18:20:37.952731447 +0000 UTC m=+4196.596700936"
Jul 28 18:20:38.934930 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:38.934894    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:38.936258 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:38.936220    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:20:39.939231 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:39.939191    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:20:44.943229 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:44.943200    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:20:44.943832 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:44.943803    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:20:54.943814 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:20:54.943776    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:21:04.944429 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:21:04.944340    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:21:14.943973 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:21:14.943929    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:21:16.075083 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:21:16.075054    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:21:16.081670 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:21:16.081641    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:21:24.943973 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:21:24.943927    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:21:34.943778 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:21:34.943738    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:21:44.944374 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:21:44.944338    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:21:54.944408 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:21:54.944374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:22:02.635305 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:02.635266    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"]
Jul 28 18:22:02.635713 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:02.635512    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" containerID="cri-o://2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a" gracePeriod=30
Jul 28 18:22:02.635713 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:02.635561    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kube-rbac-proxy" containerID="cri-o://1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a" gracePeriod=30
Jul 28 18:22:03.271373 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:03.271340    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerID="1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a" exitCode=2
Jul 28 18:22:03.271523 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:03.271412    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" event={"ID":"ef87f0ea-95b0-4180-a0e5-6f8144d4711a","Type":"ContainerDied","Data":"1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a"}
Jul 28 18:22:04.939924 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:04.939876    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 28 18:22:04.944186 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:04.944155    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 28 18:22:07.081601 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.081576    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:22:07.151341 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.151318    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-proxy-tls\") pod \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") "
Jul 28 18:22:07.151471 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.151348    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-rbac-proxy-sar-config\") pod \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") "
Jul 28 18:22:07.151471 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.151387    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kserve-provision-location\") pod \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") "
Jul 28 18:22:07.151471 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.151419    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqqrr\" (UniqueName: \"kubernetes.io/projected/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-api-access-zqqrr\") pod \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") "
Jul 28 18:22:07.151471 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.151441    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-cabundle-cert\") pod \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\" (UID: \"ef87f0ea-95b0-4180-a0e5-6f8144d4711a\") "
Jul 28 18:22:07.151779 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.151752    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef87f0ea-95b0-4180-a0e5-6f8144d4711a" (UID: "ef87f0ea-95b0-4180-a0e5-6f8144d4711a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:22:07.152385 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.152342    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef87f0ea-95b0-4180-a0e5-6f8144d4711a" (UID: "ef87f0ea-95b0-4180-a0e5-6f8144d4711a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:22:07.152385 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.151812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef87f0ea-95b0-4180-a0e5-6f8144d4711a" (UID: "ef87f0ea-95b0-4180-a0e5-6f8144d4711a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:22:07.154324 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.153530    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef87f0ea-95b0-4180-a0e5-6f8144d4711a" (UID: "ef87f0ea-95b0-4180-a0e5-6f8144d4711a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:22:07.154324 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.153589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-api-access-zqqrr" (OuterVolumeSpecName: "kube-api-access-zqqrr") pod "ef87f0ea-95b0-4180-a0e5-6f8144d4711a" (UID: "ef87f0ea-95b0-4180-a0e5-6f8144d4711a"). InnerVolumeSpecName "kube-api-access-zqqrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:22:07.252385 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.252358    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqqrr\" (UniqueName: \"kubernetes.io/projected/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-api-access-zqqrr\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:22:07.252385 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.252381    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:22:07.252571 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.252391    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:22:07.252571 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.252399    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:22:07.252571 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.252408    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef87f0ea-95b0-4180-a0e5-6f8144d4711a-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:22:07.288722 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.288692    2577 generic.go:356] "Generic (PLEG): container finished" podID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerID="2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a" exitCode=0
Jul 28 18:22:07.288832 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.288762    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"
Jul 28 18:22:07.288832 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.288769    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" event={"ID":"ef87f0ea-95b0-4180-a0e5-6f8144d4711a","Type":"ContainerDied","Data":"2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a"}
Jul 28 18:22:07.288912 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.288870    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m" event={"ID":"ef87f0ea-95b0-4180-a0e5-6f8144d4711a","Type":"ContainerDied","Data":"5d0ebb7ccd420f51f4b5a9267bb3af83e5dca1462683d99480d5b0840dfc0e22"}
Jul 28 18:22:07.288947 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.288911    2577 scope.go:117] "RemoveContainer" containerID="1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a"
Jul 28 18:22:07.297796 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.297781    2577 scope.go:117] "RemoveContainer" containerID="2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a"
Jul 28 18:22:07.304889 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.304875    2577 scope.go:117] "RemoveContainer" containerID="2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398"
Jul 28 18:22:07.311820 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.311804    2577 scope.go:117] "RemoveContainer" containerID="1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a"
Jul 28 18:22:07.312094 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:22:07.312061    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a\": container with ID starting with 1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a not found: ID does not exist" containerID="1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a"
Jul 28 18:22:07.312167 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.312102    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a"} err="failed to get container status \"1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a\": rpc error: code = NotFound desc = could not find container \"1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a\": container with ID starting with 1e107052d0059a6bd382a4f46cf5d61a780237dc1844bee8bf35c091e19ccb5a not found: ID does not exist"
Jul 28 18:22:07.312167 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.312124    2577 scope.go:117] "RemoveContainer" containerID="2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a"
Jul 28 18:22:07.312167 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.312136    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"]
Jul 28 18:22:07.312359 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:22:07.312340    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a\": container with ID starting with 2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a not found: ID does not exist" containerID="2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a"
Jul 28 18:22:07.312397 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.312366    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a"} err="failed to get container status \"2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a\": rpc error: code = NotFound desc = could not find container \"2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a\": container with ID starting with 2d51d2c3c9e6b07fd998444221db6372efac14f682c766d05a00869683491b4a not found: ID does not exist"
Jul 28 18:22:07.312397 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.312382    2577 scope.go:117] "RemoveContainer" containerID="2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398"
Jul 28 18:22:07.312595 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:22:07.312578    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398\": container with ID starting with 2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398 not found: ID does not exist" containerID="2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398"
Jul 28 18:22:07.312641 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.312600    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398"} err="failed to get container status \"2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398\": rpc error: code = NotFound desc = could not find container \"2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398\": container with ID starting with 2d3300ed5f4ab6eb2319d68bedf04bddb7ad27dfe5a39841452f529e27e91398 not found: ID does not exist"
Jul 28 18:22:07.317856 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.317826    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-66f8b44c4f-mrp7m"]
Jul 28 18:22:07.904357 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:22:07.904328    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" path="/var/lib/kubelet/pods/ef87f0ea-95b0-4180-a0e5-6f8144d4711a/volumes"
Jul 28 18:23:34.611097 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.611060    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"]
Jul 28 18:23:34.611642 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.611379    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kube-rbac-proxy"
Jul 28 18:23:34.611642 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.611390    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kube-rbac-proxy"
Jul 28 18:23:34.611642 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.611407    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="storage-initializer"
Jul 28 18:23:34.611642 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.611412    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="storage-initializer"
Jul 28 18:23:34.611642 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.611420    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container"
Jul 28 18:23:34.611642 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.611425    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container"
Jul 28 18:23:34.611642 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.611473    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kserve-container"
Jul 28 18:23:34.611642 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.611485    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef87f0ea-95b0-4180-a0e5-6f8144d4711a" containerName="kube-rbac-proxy"
Jul 28 18:23:34.614665 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.614646    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.618498 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.618464    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-predictor-serving-cert\""
Jul 28 18:23:34.618498 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.618467    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-triton-kube-rbac-proxy-sar-config\""
Jul 28 18:23:34.618654 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.618467    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:23:34.618654 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.618512    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:23:34.618654 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.618471    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:23:34.619875 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.619853    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:23:34.619875 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.619865    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:23:34.625750 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.625730    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"]
Jul 28 18:23:34.745527 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.745491    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf64b888-a906-4531-9a14-c6d09ebad97f-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.745527 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.745531    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l57ph\" (UniqueName: \"kubernetes.io/projected/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-api-access-l57ph\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.745765 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.745551    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.745765 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.745572    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf64b888-a906-4531-9a14-c6d09ebad97f-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.745765 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.745624    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.846352 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.846319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf64b888-a906-4531-9a14-c6d09ebad97f-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.846352 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.846355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57ph\" (UniqueName: \"kubernetes.io/projected/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-api-access-l57ph\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.846597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.846377    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.846597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.846405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf64b888-a906-4531-9a14-c6d09ebad97f-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.846597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.846455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.846947 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.846915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf64b888-a906-4531-9a14-c6d09ebad97f-kserve-provision-location\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.847085 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.847015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-rbac-proxy-sar-config\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.847194 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.847174    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-cabundle-cert\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.848870 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.848852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf64b888-a906-4531-9a14-c6d09ebad97f-proxy-tls\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.855781 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.855762    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57ph\" (UniqueName: \"kubernetes.io/projected/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-api-access-l57ph\") pod \"isvc-triton-predictor-5d7db58676-wztpp\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") " pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:34.925804 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:34.925771    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:23:35.053334 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:35.053309    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"]
Jul 28 18:23:35.055322 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:23:35.055295    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf64b888_a906_4531_9a14_c6d09ebad97f.slice/crio-1099ef19bd88d42e8a8f0e2ab65d784d0edfe24e01cdb508f7571242244da633 WatchSource:0}: Error finding container 1099ef19bd88d42e8a8f0e2ab65d784d0edfe24e01cdb508f7571242244da633: Status 404 returned error can't find the container with id 1099ef19bd88d42e8a8f0e2ab65d784d0edfe24e01cdb508f7571242244da633
Jul 28 18:23:35.639953 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:35.639914    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" event={"ID":"bf64b888-a906-4531-9a14-c6d09ebad97f","Type":"ContainerStarted","Data":"8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37"}
Jul 28 18:23:35.640449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:35.639983    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" event={"ID":"bf64b888-a906-4531-9a14-c6d09ebad97f","Type":"ContainerStarted","Data":"1099ef19bd88d42e8a8f0e2ab65d784d0edfe24e01cdb508f7571242244da633"}
Jul 28 18:23:39.661958 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:39.661922    2577 generic.go:356] "Generic (PLEG): container finished" podID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerID="8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37" exitCode=0
Jul 28 18:23:39.662400 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:39.661993    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" event={"ID":"bf64b888-a906-4531-9a14-c6d09ebad97f","Type":"ContainerDied","Data":"8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37"}
Jul 28 18:23:39.663179 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:23:39.663162    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:25:35.243344 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:35.243319    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" event={"ID":"bf64b888-a906-4531-9a14-c6d09ebad97f","Type":"ContainerStarted","Data":"9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6"}
Jul 28 18:25:36.249069 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:36.249031    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" event={"ID":"bf64b888-a906-4531-9a14-c6d09ebad97f","Type":"ContainerStarted","Data":"f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb"}
Jul 28 18:25:36.249475 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:36.249103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:25:36.278224 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:36.278172    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" podStartSLOduration=2.7635847780000002 podStartE2EDuration="2m2.278154717s" totalImagesPullingTime="1m55.465019632s" totalInitContainerRuntime="4.049550307s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:25:36.277602292 +0000 UTC m=+4494.921571811" watchObservedRunningTime="2026-07-28 18:25:36.278154717 +0000 UTC m=+4494.922124208"
Jul 28 18:25:37.253194 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:37.253164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:25:37.254214 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:37.254186    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 18:25:38.256947 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:38.256908    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 18:25:43.261011 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:43.260982    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:25:43.261698 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:43.261681    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:25:46.453184 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:46.453146    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"]
Jul 28 18:25:46.453614 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:46.453396    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kserve-container" containerID="cri-o://9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6" gracePeriod=30
Jul 28 18:25:46.453614 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:46.453467    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kube-rbac-proxy" containerID="cri-o://f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb" gracePeriod=30
Jul 28 18:25:47.295068 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:47.295030    2577 generic.go:356] "Generic (PLEG): container finished" podID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerID="f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb" exitCode=2
Jul 28 18:25:47.295299 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:47.295100    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" event={"ID":"bf64b888-a906-4531-9a14-c6d09ebad97f","Type":"ContainerDied","Data":"f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb"}
Jul 28 18:25:48.257900 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:48.257858    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 18:25:49.070961 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.070938    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:25:49.163155 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.163122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-rbac-proxy-sar-config\") pod \"bf64b888-a906-4531-9a14-c6d09ebad97f\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") "
Jul 28 18:25:49.163477 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.163180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-cabundle-cert\") pod \"bf64b888-a906-4531-9a14-c6d09ebad97f\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") "
Jul 28 18:25:49.163477 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.163237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf64b888-a906-4531-9a14-c6d09ebad97f-proxy-tls\") pod \"bf64b888-a906-4531-9a14-c6d09ebad97f\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") "
Jul 28 18:25:49.163477 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.163276    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf64b888-a906-4531-9a14-c6d09ebad97f-kserve-provision-location\") pod \"bf64b888-a906-4531-9a14-c6d09ebad97f\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") "
Jul 28 18:25:49.163477 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.163304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l57ph\" (UniqueName: \"kubernetes.io/projected/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-api-access-l57ph\") pod \"bf64b888-a906-4531-9a14-c6d09ebad97f\" (UID: \"bf64b888-a906-4531-9a14-c6d09ebad97f\") "
Jul 28 18:25:49.163690 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.163598    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bf64b888-a906-4531-9a14-c6d09ebad97f" (UID: "bf64b888-a906-4531-9a14-c6d09ebad97f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:25:49.163746 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.163695    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bf64b888-a906-4531-9a14-c6d09ebad97f" (UID: "bf64b888-a906-4531-9a14-c6d09ebad97f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:25:49.163784 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.163741    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf64b888-a906-4531-9a14-c6d09ebad97f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bf64b888-a906-4531-9a14-c6d09ebad97f" (UID: "bf64b888-a906-4531-9a14-c6d09ebad97f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:25:49.165446 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.165423    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf64b888-a906-4531-9a14-c6d09ebad97f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bf64b888-a906-4531-9a14-c6d09ebad97f" (UID: "bf64b888-a906-4531-9a14-c6d09ebad97f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:25:49.165548 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.165477    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-api-access-l57ph" (OuterVolumeSpecName: "kube-api-access-l57ph") pod "bf64b888-a906-4531-9a14-c6d09ebad97f" (UID: "bf64b888-a906-4531-9a14-c6d09ebad97f"). InnerVolumeSpecName "kube-api-access-l57ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:25:49.264356 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.264285    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l57ph\" (UniqueName: \"kubernetes.io/projected/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-api-access-l57ph\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:25:49.264356 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.264309    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:25:49.264356 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.264319    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bf64b888-a906-4531-9a14-c6d09ebad97f-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:25:49.264356 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.264328    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bf64b888-a906-4531-9a14-c6d09ebad97f-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:25:49.264356 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.264335    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bf64b888-a906-4531-9a14-c6d09ebad97f-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:25:49.304694 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.304665    2577 generic.go:356] "Generic (PLEG): container finished" podID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerID="9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6" exitCode=0
Jul 28 18:25:49.304878 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.304726    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"
Jul 28 18:25:49.304878 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.304735    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" event={"ID":"bf64b888-a906-4531-9a14-c6d09ebad97f","Type":"ContainerDied","Data":"9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6"}
Jul 28 18:25:49.304878 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.304798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp" event={"ID":"bf64b888-a906-4531-9a14-c6d09ebad97f","Type":"ContainerDied","Data":"1099ef19bd88d42e8a8f0e2ab65d784d0edfe24e01cdb508f7571242244da633"}
Jul 28 18:25:49.304878 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.304838    2577 scope.go:117] "RemoveContainer" containerID="f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb"
Jul 28 18:25:49.313953 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.313933    2577 scope.go:117] "RemoveContainer" containerID="9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6"
Jul 28 18:25:49.321197 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.321179    2577 scope.go:117] "RemoveContainer" containerID="8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37"
Jul 28 18:25:49.327513 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.327491    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"]
Jul 28 18:25:49.328297 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.328284    2577 scope.go:117] "RemoveContainer" containerID="f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb"
Jul 28 18:25:49.328556 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:25:49.328535    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb\": container with ID starting with f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb not found: ID does not exist" containerID="f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb"
Jul 28 18:25:49.328596 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.328565    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb"} err="failed to get container status \"f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb\": rpc error: code = NotFound desc = could not find container \"f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb\": container with ID starting with f1836e5046635a7b3b3d6189ad1f29e228016f34956d82640d765e2dd9076feb not found: ID does not exist"
Jul 28 18:25:49.328596 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.328583    2577 scope.go:117] "RemoveContainer" containerID="9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6"
Jul 28 18:25:49.328813 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:25:49.328795    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6\": container with ID starting with 9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6 not found: ID does not exist" containerID="9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6"
Jul 28 18:25:49.328860 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.328819    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6"} err="failed to get container status \"9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6\": rpc error: code = NotFound desc = could not find container \"9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6\": container with ID starting with 9cdbbb4f62e9e67a06ae782da6b2c535fefe363071b7ec8c99e315231a7ed2c6 not found: ID does not exist"
Jul 28 18:25:49.328860 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.328837    2577 scope.go:117] "RemoveContainer" containerID="8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37"
Jul 28 18:25:49.329043 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:25:49.329027    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37\": container with ID starting with 8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37 not found: ID does not exist" containerID="8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37"
Jul 28 18:25:49.329102 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.329047    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37"} err="failed to get container status \"8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37\": rpc error: code = NotFound desc = could not find container \"8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37\": container with ID starting with 8c7fb34808e6b6cae526e82e786b71015a891cff18ef1d6bc064495a96266f37 not found: ID does not exist"
Jul 28 18:25:49.331414 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.331394    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-triton-predictor-5d7db58676-wztpp"]
Jul 28 18:25:49.904962 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:49.904928    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" path="/var/lib/kubelet/pods/bf64b888-a906-4531-9a14-c6d09ebad97f/volumes"
Jul 28 18:25:51.659799 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.659767    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"]
Jul 28 18:25:51.660185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.660067    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kube-rbac-proxy"
Jul 28 18:25:51.660185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.660077    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kube-rbac-proxy"
Jul 28 18:25:51.660185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.660091    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kserve-container"
Jul 28 18:25:51.660185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.660096    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kserve-container"
Jul 28 18:25:51.660185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.660114    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="storage-initializer"
Jul 28 18:25:51.660185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.660120    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="storage-initializer"
Jul 28 18:25:51.660185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.660165    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kube-rbac-proxy"
Jul 28 18:25:51.660185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.660176    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf64b888-a906-4531-9a14-c6d09ebad97f" containerName="kserve-container"
Jul 28 18:25:51.688185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.688121    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"]
Jul 28 18:25:51.688387 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.688310    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.692295 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.692272    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 28 18:25:51.692432 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.692351    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:25:51.692976 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.692933    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:25:51.693093 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.692990    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:25:51.693093 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.692991    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:25:51.693472 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.693453    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 28 18:25:51.693586 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.693535    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:25:51.784311 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.784272    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb298642-d3dd-4e05-b368-43764c141918-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.784507 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.784319    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.784507 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.784374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtls\" (UniqueName: \"kubernetes.io/projected/eb298642-d3dd-4e05-b368-43764c141918-kube-api-access-cvtls\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.784507 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.784457    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb298642-d3dd-4e05-b368-43764c141918-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.784507 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.784497    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.885193 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.885152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtls\" (UniqueName: \"kubernetes.io/projected/eb298642-d3dd-4e05-b368-43764c141918-kube-api-access-cvtls\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.885380 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.885202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb298642-d3dd-4e05-b368-43764c141918-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.885380 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.885240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.885380 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.885328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb298642-d3dd-4e05-b368-43764c141918-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.885380 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.885356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.885822 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.885799    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb298642-d3dd-4e05-b368-43764c141918-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.886102 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.886084    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.886185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.886167    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.887765 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.887743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb298642-d3dd-4e05-b368-43764c141918-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.895692 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.895669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtls\" (UniqueName: \"kubernetes.io/projected/eb298642-d3dd-4e05-b368-43764c141918-kube-api-access-cvtls\") pod \"isvc-xgboost-predictor-7566d65bf7-6mwxw\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:51.999555 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:51.999466    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:25:52.174698 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:52.172387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"]
Jul 28 18:25:52.175638 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:25:52.175612    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb298642_d3dd_4e05_b368_43764c141918.slice/crio-b436688417e4f23f1f4f5b120453aa66100c45f3e872486d86988362f1c22a94 WatchSource:0}: Error finding container b436688417e4f23f1f4f5b120453aa66100c45f3e872486d86988362f1c22a94: Status 404 returned error can't find the container with id b436688417e4f23f1f4f5b120453aa66100c45f3e872486d86988362f1c22a94
Jul 28 18:25:52.319197 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:52.319166    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" event={"ID":"eb298642-d3dd-4e05-b368-43764c141918","Type":"ContainerStarted","Data":"2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c"}
Jul 28 18:25:52.319551 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:52.319232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" event={"ID":"eb298642-d3dd-4e05-b368-43764c141918","Type":"ContainerStarted","Data":"b436688417e4f23f1f4f5b120453aa66100c45f3e872486d86988362f1c22a94"}
Jul 28 18:25:56.337319 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:56.337287    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb298642-d3dd-4e05-b368-43764c141918" containerID="2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c" exitCode=0
Jul 28 18:25:56.337702 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:25:56.337371    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" event={"ID":"eb298642-d3dd-4e05-b368-43764c141918","Type":"ContainerDied","Data":"2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c"}
Jul 28 18:26:16.103026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:16.102998    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:26:16.109026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:16.109005    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:26:16.431464 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:16.431429    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" event={"ID":"eb298642-d3dd-4e05-b368-43764c141918","Type":"ContainerStarted","Data":"f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449"}
Jul 28 18:26:16.431629 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:16.431496    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" event={"ID":"eb298642-d3dd-4e05-b368-43764c141918","Type":"ContainerStarted","Data":"bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3"}
Jul 28 18:26:16.431711 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:16.431688    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:26:16.453817 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:16.453772    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podStartSLOduration=2.330905077 podStartE2EDuration="25.453757363s" totalImagesPullingTime="19.098218583s" totalInitContainerRuntime="4.024633703s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:26:16.452012932 +0000 UTC m=+4535.095982432" watchObservedRunningTime="2026-07-28 18:26:16.453757363 +0000 UTC m=+4535.097726854"
Jul 28 18:26:17.435577 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:17.435539    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:26:17.437014 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:17.436983    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 18:26:18.439982 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:18.439938    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 18:26:23.445097 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:23.445071    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:26:23.445665 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:23.445538    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 18:26:33.446101 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:33.446063    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 18:26:43.446324 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:43.446284    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 18:26:53.446180 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:26:53.446140    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 18:27:03.446302 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:03.446208    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 18:27:13.446379 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:13.446352    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:27:21.740141 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:21.740105    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"]
Jul 28 18:27:21.740568 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:21.740410    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container" containerID="cri-o://bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3" gracePeriod=30
Jul 28 18:27:21.740568 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:21.740451    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kube-rbac-proxy" containerID="cri-o://f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449" gracePeriod=30
Jul 28 18:27:22.695826 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:22.695794    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb298642-d3dd-4e05-b368-43764c141918" containerID="f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449" exitCode=2
Jul 28 18:27:22.695988 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:22.695865    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" event={"ID":"eb298642-d3dd-4e05-b368-43764c141918","Type":"ContainerDied","Data":"f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449"}
Jul 28 18:27:23.440626 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:23.440587    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 28 18:27:23.445935 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:23.445907    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 18:27:24.976426 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:24.976404    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:27:25.044687 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.044660    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb298642-d3dd-4e05-b368-43764c141918-proxy-tls\") pod \"eb298642-d3dd-4e05-b368-43764c141918\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") "
Jul 28 18:27:25.044816 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.044697    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb298642-d3dd-4e05-b368-43764c141918-kserve-provision-location\") pod \"eb298642-d3dd-4e05-b368-43764c141918\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") "
Jul 28 18:27:25.044816 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.044729    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvtls\" (UniqueName: \"kubernetes.io/projected/eb298642-d3dd-4e05-b368-43764c141918-kube-api-access-cvtls\") pod \"eb298642-d3dd-4e05-b368-43764c141918\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") "
Jul 28 18:27:25.044938 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.044860    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-kube-rbac-proxy-sar-config\") pod \"eb298642-d3dd-4e05-b368-43764c141918\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") "
Jul 28 18:27:25.044938 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.044930    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-cabundle-cert\") pod \"eb298642-d3dd-4e05-b368-43764c141918\" (UID: \"eb298642-d3dd-4e05-b368-43764c141918\") "
Jul 28 18:27:25.045044 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.044996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb298642-d3dd-4e05-b368-43764c141918-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb298642-d3dd-4e05-b368-43764c141918" (UID: "eb298642-d3dd-4e05-b368-43764c141918"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:27:25.045160 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.045144    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb298642-d3dd-4e05-b368-43764c141918-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:27:25.045218 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.045181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb298642-d3dd-4e05-b368-43764c141918" (UID: "eb298642-d3dd-4e05-b368-43764c141918"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:27:25.045283 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.045264    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb298642-d3dd-4e05-b368-43764c141918" (UID: "eb298642-d3dd-4e05-b368-43764c141918"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:27:25.047206 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.047187    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb298642-d3dd-4e05-b368-43764c141918-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb298642-d3dd-4e05-b368-43764c141918" (UID: "eb298642-d3dd-4e05-b368-43764c141918"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:27:25.047382 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.047218    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb298642-d3dd-4e05-b368-43764c141918-kube-api-access-cvtls" (OuterVolumeSpecName: "kube-api-access-cvtls") pod "eb298642-d3dd-4e05-b368-43764c141918" (UID: "eb298642-d3dd-4e05-b368-43764c141918"). InnerVolumeSpecName "kube-api-access-cvtls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:27:25.145652 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.145632    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:27:25.145652 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.145653    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb298642-d3dd-4e05-b368-43764c141918-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:27:25.145797 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.145662    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvtls\" (UniqueName: \"kubernetes.io/projected/eb298642-d3dd-4e05-b368-43764c141918-kube-api-access-cvtls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:27:25.145797 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.145671    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb298642-d3dd-4e05-b368-43764c141918-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:27:25.709773 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.709738    2577 generic.go:356] "Generic (PLEG): container finished" podID="eb298642-d3dd-4e05-b368-43764c141918" containerID="bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3" exitCode=0
Jul 28 18:27:25.709951 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.709787    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"
Jul 28 18:27:25.709951 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.709856    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" event={"ID":"eb298642-d3dd-4e05-b368-43764c141918","Type":"ContainerDied","Data":"bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3"}
Jul 28 18:27:25.709951 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.709923    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw" event={"ID":"eb298642-d3dd-4e05-b368-43764c141918","Type":"ContainerDied","Data":"b436688417e4f23f1f4f5b120453aa66100c45f3e872486d86988362f1c22a94"}
Jul 28 18:27:25.710096 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.709962    2577 scope.go:117] "RemoveContainer" containerID="f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449"
Jul 28 18:27:25.718580 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.718564    2577 scope.go:117] "RemoveContainer" containerID="bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3"
Jul 28 18:27:25.725891 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.725876    2577 scope.go:117] "RemoveContainer" containerID="2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c"
Jul 28 18:27:25.734282 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.733328    2577 scope.go:117] "RemoveContainer" containerID="f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449"
Jul 28 18:27:25.735470 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:27:25.735447    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449\": container with ID starting with f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449 not found: ID does not exist" containerID="f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449"
Jul 28 18:27:25.735562 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.735475    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449"} err="failed to get container status \"f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449\": rpc error: code = NotFound desc = could not find container \"f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449\": container with ID starting with f83646d7f453ea2107a08ed5ba10ae3e5012d5205ab149bd58f583f964b7f449 not found: ID does not exist"
Jul 28 18:27:25.735562 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.735491    2577 scope.go:117] "RemoveContainer" containerID="bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3"
Jul 28 18:27:25.735812 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:27:25.735794    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3\": container with ID starting with bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3 not found: ID does not exist" containerID="bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3"
Jul 28 18:27:25.735886 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.735816    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3"} err="failed to get container status \"bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3\": rpc error: code = NotFound desc = could not find container \"bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3\": container with ID starting with bfd7a567b4c09f8fbb1817eaa40328e2da2149e99ba72c51850168ef502b14d3 not found: ID does not exist"
Jul 28 18:27:25.735886 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.735832    2577 scope.go:117] "RemoveContainer" containerID="2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c"
Jul 28 18:27:25.735977 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.735933    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"]
Jul 28 18:27:25.736084 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:27:25.736065    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c\": container with ID starting with 2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c not found: ID does not exist" containerID="2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c"
Jul 28 18:27:25.736126 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.736093    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c"} err="failed to get container status \"2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c\": rpc error: code = NotFound desc = could not find container \"2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c\": container with ID starting with 2e4559761b80e4468fddc6b00cb3977d8ad5f8383d3e9e5f753487439067595c not found: ID does not exist"
Jul 28 18:27:25.741102 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.741081    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-6mwxw"]
Jul 28 18:27:25.904894 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:25.904866    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb298642-d3dd-4e05-b368-43764c141918" path="/var/lib/kubelet/pods/eb298642-d3dd-4e05-b368-43764c141918/volumes"
Jul 28 18:27:26.989319 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.989290    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"]
Jul 28 18:27:26.989697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.989591    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="storage-initializer"
Jul 28 18:27:26.989697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.989602    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="storage-initializer"
Jul 28 18:27:26.989697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.989610    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container"
Jul 28 18:27:26.989697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.989615    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container"
Jul 28 18:27:26.989697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.989622    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kube-rbac-proxy"
Jul 28 18:27:26.989697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.989628    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kube-rbac-proxy"
Jul 28 18:27:26.989697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.989673    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kube-rbac-proxy"
Jul 28 18:27:26.989697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.989681    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb298642-d3dd-4e05-b368-43764c141918" containerName="kserve-container"
Jul 28 18:27:26.992927 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.992903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:26.996106 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.996087    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:27:26.996216 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.996105    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:27:26.997432 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.997414    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:27:26.997542 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.997449    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:27:26.997542 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.997471    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:27:26.997542 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.997452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 18:27:26.997542 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:26.997424    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 18:27:27.005596 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.005572    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"]
Jul 28 18:27:27.062067 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.062037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.062195 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.062081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p7xm\" (UniqueName: \"kubernetes.io/projected/a4da4a41-6e26-4a25-8598-60a55743396c-kube-api-access-8p7xm\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.062195 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.062121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4da4a41-6e26-4a25-8598-60a55743396c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.062195 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.062143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.062195 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.062173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4da4a41-6e26-4a25-8598-60a55743396c-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.163480 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.163443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p7xm\" (UniqueName: \"kubernetes.io/projected/a4da4a41-6e26-4a25-8598-60a55743396c-kube-api-access-8p7xm\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.163597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.163492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4da4a41-6e26-4a25-8598-60a55743396c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.163597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.163511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.163597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.163545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4da4a41-6e26-4a25-8598-60a55743396c-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.163597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.163566    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.163964 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.163943    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4da4a41-6e26-4a25-8598-60a55743396c-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.164222 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.164202    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.164325 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.164274    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.165876 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.165860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4da4a41-6e26-4a25-8598-60a55743396c-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.172762 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.172742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p7xm\" (UniqueName: \"kubernetes.io/projected/a4da4a41-6e26-4a25-8598-60a55743396c-kube-api-access-8p7xm\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.303048 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.302987    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:27.429458 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.429433    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"]
Jul 28 18:27:27.432011 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:27:27.431984    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4da4a41_6e26_4a25_8598_60a55743396c.slice/crio-ba267e970e7ecc111439acd50730749b17dec802385ad53ef6ff071831a286de WatchSource:0}: Error finding container ba267e970e7ecc111439acd50730749b17dec802385ad53ef6ff071831a286de: Status 404 returned error can't find the container with id ba267e970e7ecc111439acd50730749b17dec802385ad53ef6ff071831a286de
Jul 28 18:27:27.719766 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.719728    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" event={"ID":"a4da4a41-6e26-4a25-8598-60a55743396c","Type":"ContainerStarted","Data":"cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd"}
Jul 28 18:27:27.719961 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:27.719797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" event={"ID":"a4da4a41-6e26-4a25-8598-60a55743396c","Type":"ContainerStarted","Data":"ba267e970e7ecc111439acd50730749b17dec802385ad53ef6ff071831a286de"}
Jul 28 18:27:31.737048 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:31.737018    2577 generic.go:356] "Generic (PLEG): container finished" podID="a4da4a41-6e26-4a25-8598-60a55743396c" containerID="cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd" exitCode=0
Jul 28 18:27:31.737426 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:31.737093    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" event={"ID":"a4da4a41-6e26-4a25-8598-60a55743396c","Type":"ContainerDied","Data":"cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd"}
Jul 28 18:27:32.744151 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:32.744118    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" event={"ID":"a4da4a41-6e26-4a25-8598-60a55743396c","Type":"ContainerStarted","Data":"3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245"}
Jul 28 18:27:32.744565 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:32.744180    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" event={"ID":"a4da4a41-6e26-4a25-8598-60a55743396c","Type":"ContainerStarted","Data":"f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8"}
Jul 28 18:27:32.744565 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:32.744413    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:32.744659 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:32.744575    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:27:32.767179 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:32.767139    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" podStartSLOduration=2.944048798 podStartE2EDuration="6.767127503s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.823078705s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:27:32.765509435 +0000 UTC m=+4611.409478949" watchObservedRunningTime="2026-07-28 18:27:32.767127503 +0000 UTC m=+4611.411096993"
Jul 28 18:27:38.756000 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:27:38.755973    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:28:08.759649 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:08.759623    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:28:16.999590 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:16.999554    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"]
Jul 28 18:28:17.000078 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:16.999852    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="kserve-container" containerID="cri-o://f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8" gracePeriod=30
Jul 28 18:28:17.000078 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:16.999917    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="kube-rbac-proxy" containerID="cri-o://3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245" gracePeriod=30
Jul 28 18:28:17.933843 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:17.933811    2577 generic.go:356] "Generic (PLEG): container finished" podID="a4da4a41-6e26-4a25-8598-60a55743396c" containerID="3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245" exitCode=2
Jul 28 18:28:17.934045 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:17.933860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" event={"ID":"a4da4a41-6e26-4a25-8598-60a55743396c","Type":"ContainerDied","Data":"3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245"}
Jul 28 18:28:18.750872 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:18.750829    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 28 18:28:22.184592 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.184559    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"]
Jul 28 18:28:22.188099 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.188080    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.191113 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.191084    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jul 28 18:28:22.191113 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.191101    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jul 28 18:28:22.198877 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.198855    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"]
Jul 28 18:28:22.287948 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.287923    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.288098 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.287961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.288098 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.287991    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-api-access-x47nt\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.288098 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.288080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/325e2418-1a48-4829-b8be-bb3d4a9b90f3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.288222 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.288111    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.388441 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.388417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.388575 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.388452    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.388575 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.388482    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.388575 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.388523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-api-access-x47nt\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.388734 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.388575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/325e2418-1a48-4829-b8be-bb3d4a9b90f3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.388797 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:28:22.388743    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 28 18:28:22.388855 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:28:22.388805    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/325e2418-1a48-4829-b8be-bb3d4a9b90f3-proxy-tls podName:325e2418-1a48-4829-b8be-bb3d4a9b90f3 nodeName:}" failed. No retries permitted until 2026-07-28 18:28:22.888786736 +0000 UTC m=+4661.532756206 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/325e2418-1a48-4829-b8be-bb3d4a9b90f3-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" (UID: "325e2418-1a48-4829-b8be-bb3d4a9b90f3") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Jul 28 18:28:22.388925 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.388856    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.389133 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.389111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.389259 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.389225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.397550 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.397527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-api-access-x47nt\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.893086 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.893058    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/325e2418-1a48-4829-b8be-bb3d4a9b90f3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:22.895215 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:22.895193    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/325e2418-1a48-4829-b8be-bb3d4a9b90f3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:23.097916 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.097882    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:23.237818 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.237785    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"]
Jul 28 18:28:23.238108 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.237917    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:28:23.239037 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:28:23.239016    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325e2418_1a48_4829_b8be_bb3d4a9b90f3.slice/crio-97a85f15e8049f065e854da6d5aa5f55a8711afdfd0741f1b777ca5cf13317c4 WatchSource:0}: Error finding container 97a85f15e8049f065e854da6d5aa5f55a8711afdfd0741f1b777ca5cf13317c4: Status 404 returned error can't find the container with id 97a85f15e8049f065e854da6d5aa5f55a8711afdfd0741f1b777ca5cf13317c4
Jul 28 18:28:23.296430 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.296407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4da4a41-6e26-4a25-8598-60a55743396c-kserve-provision-location\") pod \"a4da4a41-6e26-4a25-8598-60a55743396c\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") "
Jul 28 18:28:23.296572 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.296484    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4da4a41-6e26-4a25-8598-60a55743396c-proxy-tls\") pod \"a4da4a41-6e26-4a25-8598-60a55743396c\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") "
Jul 28 18:28:23.296572 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.296535    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p7xm\" (UniqueName: \"kubernetes.io/projected/a4da4a41-6e26-4a25-8598-60a55743396c-kube-api-access-8p7xm\") pod \"a4da4a41-6e26-4a25-8598-60a55743396c\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") "
Jul 28 18:28:23.296759 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.296593    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-cabundle-cert\") pod \"a4da4a41-6e26-4a25-8598-60a55743396c\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") "
Jul 28 18:28:23.296759 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.296622    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-kube-rbac-proxy-sar-config\") pod \"a4da4a41-6e26-4a25-8598-60a55743396c\" (UID: \"a4da4a41-6e26-4a25-8598-60a55743396c\") "
Jul 28 18:28:23.296759 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.296704    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4da4a41-6e26-4a25-8598-60a55743396c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4da4a41-6e26-4a25-8598-60a55743396c" (UID: "a4da4a41-6e26-4a25-8598-60a55743396c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:28:23.296954 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.296931    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4da4a41-6e26-4a25-8598-60a55743396c-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:28:23.297068 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.296961    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4da4a41-6e26-4a25-8598-60a55743396c" (UID: "a4da4a41-6e26-4a25-8598-60a55743396c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:28:23.297068 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.296964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4da4a41-6e26-4a25-8598-60a55743396c" (UID: "a4da4a41-6e26-4a25-8598-60a55743396c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:28:23.298373 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.298354    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4da4a41-6e26-4a25-8598-60a55743396c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4da4a41-6e26-4a25-8598-60a55743396c" (UID: "a4da4a41-6e26-4a25-8598-60a55743396c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:28:23.298881 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.298859    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4da4a41-6e26-4a25-8598-60a55743396c-kube-api-access-8p7xm" (OuterVolumeSpecName: "kube-api-access-8p7xm") pod "a4da4a41-6e26-4a25-8598-60a55743396c" (UID: "a4da4a41-6e26-4a25-8598-60a55743396c"). InnerVolumeSpecName "kube-api-access-8p7xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:28:23.397718 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.397689    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:28:23.397718 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.397713    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4da4a41-6e26-4a25-8598-60a55743396c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:28:23.397718 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.397723    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4da4a41-6e26-4a25-8598-60a55743396c-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:28:23.397925 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.397732    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p7xm\" (UniqueName: \"kubernetes.io/projected/a4da4a41-6e26-4a25-8598-60a55743396c-kube-api-access-8p7xm\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:28:23.965969 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.965933    2577 generic.go:356] "Generic (PLEG): container finished" podID="a4da4a41-6e26-4a25-8598-60a55743396c" containerID="f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8" exitCode=0
Jul 28 18:28:23.966129 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.965993    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"
Jul 28 18:28:23.966129 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.966014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" event={"ID":"a4da4a41-6e26-4a25-8598-60a55743396c","Type":"ContainerDied","Data":"f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8"}
Jul 28 18:28:23.966129 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.966073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv" event={"ID":"a4da4a41-6e26-4a25-8598-60a55743396c","Type":"ContainerDied","Data":"ba267e970e7ecc111439acd50730749b17dec802385ad53ef6ff071831a286de"}
Jul 28 18:28:23.966129 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.966113    2577 scope.go:117] "RemoveContainer" containerID="3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245"
Jul 28 18:28:23.967712 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.967683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" event={"ID":"325e2418-1a48-4829-b8be-bb3d4a9b90f3","Type":"ContainerStarted","Data":"284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783"}
Jul 28 18:28:23.967862 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.967737    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" event={"ID":"325e2418-1a48-4829-b8be-bb3d4a9b90f3","Type":"ContainerStarted","Data":"97a85f15e8049f065e854da6d5aa5f55a8711afdfd0741f1b777ca5cf13317c4"}
Jul 28 18:28:23.974292 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.974277    2577 scope.go:117] "RemoveContainer" containerID="f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8"
Jul 28 18:28:23.981421 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.981395    2577 scope.go:117] "RemoveContainer" containerID="cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd"
Jul 28 18:28:23.988309 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.988237    2577 scope.go:117] "RemoveContainer" containerID="3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245"
Jul 28 18:28:23.988552 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:28:23.988533    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245\": container with ID starting with 3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245 not found: ID does not exist" containerID="3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245"
Jul 28 18:28:23.988624 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.988559    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245"} err="failed to get container status \"3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245\": rpc error: code = NotFound desc = could not find container \"3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245\": container with ID starting with 3c8491143c7d2fcd657ce81a1c4175dbc8c461655c4bb085484bbc220f158245 not found: ID does not exist"
Jul 28 18:28:23.988624 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.988576    2577 scope.go:117] "RemoveContainer" containerID="f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8"
Jul 28 18:28:23.988804 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:28:23.988785    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8\": container with ID starting with f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8 not found: ID does not exist" containerID="f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8"
Jul 28 18:28:23.988875 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.988810    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8"} err="failed to get container status \"f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8\": rpc error: code = NotFound desc = could not find container \"f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8\": container with ID starting with f246c66b9fe27b9b7bddf817d07b6af895a0859d07f7bcabd53a1358e679cbe8 not found: ID does not exist"
Jul 28 18:28:23.988875 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.988825    2577 scope.go:117] "RemoveContainer" containerID="cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd"
Jul 28 18:28:23.989011 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:28:23.988996    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd\": container with ID starting with cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd not found: ID does not exist" containerID="cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd"
Jul 28 18:28:23.989052 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:23.989015    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd"} err="failed to get container status \"cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd\": rpc error: code = NotFound desc = could not find container \"cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd\": container with ID starting with cf77863e5833d3dc025313602676ba9f367d4e25a0ba7ea816d7ac1357759bcd not found: ID does not exist"
Jul 28 18:28:24.011074 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:24.011047    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"]
Jul 28 18:28:24.012630 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:24.012607    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-wtmqv"]
Jul 28 18:28:25.903940 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:25.903910    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" path="/var/lib/kubelet/pods/a4da4a41-6e26-4a25-8598-60a55743396c/volumes"
Jul 28 18:28:27.985524 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:27.985491    2577 generic.go:356] "Generic (PLEG): container finished" podID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerID="284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783" exitCode=0
Jul 28 18:28:27.985524 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:27.985525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" event={"ID":"325e2418-1a48-4829-b8be-bb3d4a9b90f3","Type":"ContainerDied","Data":"284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783"}
Jul 28 18:28:28.992181 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:28.992149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" event={"ID":"325e2418-1a48-4829-b8be-bb3d4a9b90f3","Type":"ContainerStarted","Data":"8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e"}
Jul 28 18:28:28.992613 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:28.992209    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" event={"ID":"325e2418-1a48-4829-b8be-bb3d4a9b90f3","Type":"ContainerStarted","Data":"a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb"}
Jul 28 18:28:28.992613 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:28.992547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:28.992613 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:28.992590    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:28:29.014051 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:29.014016    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" podStartSLOduration=3.09897874 podStartE2EDuration="7.014004287s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.915025547s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:29.01246605 +0000 UTC m=+4667.656435599" watchObservedRunningTime="2026-07-28 18:28:29.014004287 +0000 UTC m=+4667.657973776"
Jul 28 18:28:35.001407 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:28:35.001331    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:29:05.005383 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:05.005353    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:29:12.300897 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:12.300858    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"]
Jul 28 18:29:12.301412 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:12.301190    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kserve-container" containerID="cri-o://a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb" gracePeriod=30
Jul 28 18:29:12.301412 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:12.301238    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kube-rbac-proxy" containerID="cri-o://8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e" gracePeriod=30
Jul 28 18:29:13.171615 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:13.171581    2577 generic.go:356] "Generic (PLEG): container finished" podID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerID="8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e" exitCode=2
Jul 28 18:29:13.171795 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:13.171660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" event={"ID":"325e2418-1a48-4829-b8be-bb3d4a9b90f3","Type":"ContainerDied","Data":"8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e"}
Jul 28 18:29:14.996792 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:14.996747    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 28 18:29:15.002565 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:15.002532    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.43:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 28 18:29:18.450540 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.450511    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:29:18.557160 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557080    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-api-access-x47nt\") pod \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") "
Jul 28 18:29:18.557160 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557143    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-rbac-proxy-sar-config\") pod \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") "
Jul 28 18:29:18.557160 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/325e2418-1a48-4829-b8be-bb3d4a9b90f3-proxy-tls\") pod \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") "
Jul 28 18:29:18.557469 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557301    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kserve-provision-location\") pod \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") "
Jul 28 18:29:18.557469 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557373    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-cabundle-cert\") pod \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\" (UID: \"325e2418-1a48-4829-b8be-bb3d4a9b90f3\") "
Jul 28 18:29:18.557569 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557535    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "325e2418-1a48-4829-b8be-bb3d4a9b90f3" (UID: "325e2418-1a48-4829-b8be-bb3d4a9b90f3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:29:18.557623 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557576    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "325e2418-1a48-4829-b8be-bb3d4a9b90f3" (UID: "325e2418-1a48-4829-b8be-bb3d4a9b90f3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:29:18.557710 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557695    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:29:18.557750 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557713    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:29:18.557792 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.557770    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "325e2418-1a48-4829-b8be-bb3d4a9b90f3" (UID: "325e2418-1a48-4829-b8be-bb3d4a9b90f3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:29:18.559218 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.559191    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325e2418-1a48-4829-b8be-bb3d4a9b90f3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "325e2418-1a48-4829-b8be-bb3d4a9b90f3" (UID: "325e2418-1a48-4829-b8be-bb3d4a9b90f3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:29:18.559327 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.559212    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-api-access-x47nt" (OuterVolumeSpecName: "kube-api-access-x47nt") pod "325e2418-1a48-4829-b8be-bb3d4a9b90f3" (UID: "325e2418-1a48-4829-b8be-bb3d4a9b90f3"). InnerVolumeSpecName "kube-api-access-x47nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:29:18.658915 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.658882    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/325e2418-1a48-4829-b8be-bb3d4a9b90f3-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:29:18.658915 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.658908    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/325e2418-1a48-4829-b8be-bb3d4a9b90f3-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:29:18.658915 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:18.658918    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47nt\" (UniqueName: \"kubernetes.io/projected/325e2418-1a48-4829-b8be-bb3d4a9b90f3-kube-api-access-x47nt\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:29:19.197266 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.197212    2577 generic.go:356] "Generic (PLEG): container finished" podID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerID="a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb" exitCode=0
Jul 28 18:29:19.197446 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.197294    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"
Jul 28 18:29:19.197446 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.197299    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" event={"ID":"325e2418-1a48-4829-b8be-bb3d4a9b90f3","Type":"ContainerDied","Data":"a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb"}
Jul 28 18:29:19.197446 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.197359    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8" event={"ID":"325e2418-1a48-4829-b8be-bb3d4a9b90f3","Type":"ContainerDied","Data":"97a85f15e8049f065e854da6d5aa5f55a8711afdfd0741f1b777ca5cf13317c4"}
Jul 28 18:29:19.197446 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.197386    2577 scope.go:117] "RemoveContainer" containerID="8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e"
Jul 28 18:29:19.205699 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.205684    2577 scope.go:117] "RemoveContainer" containerID="a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb"
Jul 28 18:29:19.212842 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.212827    2577 scope.go:117] "RemoveContainer" containerID="284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783"
Jul 28 18:29:19.218743 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.218720    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"]
Jul 28 18:29:19.220062 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.220048    2577 scope.go:117] "RemoveContainer" containerID="8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e"
Jul 28 18:29:19.220293 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:29:19.220275    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e\": container with ID starting with 8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e not found: ID does not exist" containerID="8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e"
Jul 28 18:29:19.220372 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.220303    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e"} err="failed to get container status \"8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e\": rpc error: code = NotFound desc = could not find container \"8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e\": container with ID starting with 8de6364e6f88c868aeffc300d4a3243eeb2a3c280feb1f3b5945c7cec145b95e not found: ID does not exist"
Jul 28 18:29:19.220372 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.220324    2577 scope.go:117] "RemoveContainer" containerID="a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb"
Jul 28 18:29:19.220551 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:29:19.220534    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb\": container with ID starting with a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb not found: ID does not exist" containerID="a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb"
Jul 28 18:29:19.220600 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.220562    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb"} err="failed to get container status \"a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb\": rpc error: code = NotFound desc = could not find container \"a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb\": container with ID starting with a576441e7099ab54ae2351f317f71bad92bda1504218768e32d0edaddd17b8fb not found: ID does not exist"
Jul 28 18:29:19.220600 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.220583    2577 scope.go:117] "RemoveContainer" containerID="284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783"
Jul 28 18:29:19.220792 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:29:19.220777    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783\": container with ID starting with 284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783 not found: ID does not exist" containerID="284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783"
Jul 28 18:29:19.220834 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.220795    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783"} err="failed to get container status \"284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783\": rpc error: code = NotFound desc = could not find container \"284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783\": container with ID starting with 284f1df627dba3979a677a4e61042e90bd46920d8c04c264d3dc8e315a5fd783 not found: ID does not exist"
Jul 28 18:29:19.224116 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.224098    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-rsxm8"]
Jul 28 18:29:19.903919 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:29:19.903878    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" path="/var/lib/kubelet/pods/325e2418-1a48-4829-b8be-bb3d4a9b90f3/volumes"
Jul 28 18:30:52.818992 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.818962    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"]
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819299    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="storage-initializer"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819315    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="storage-initializer"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819326    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="kserve-container"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819334    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="kserve-container"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819350    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kserve-container"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819356    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kserve-container"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819364    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kube-rbac-proxy"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819369    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kube-rbac-proxy"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819377    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="kube-rbac-proxy"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819385    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="kube-rbac-proxy"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819400    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="storage-initializer"
Jul 28 18:30:52.819406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819405    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="storage-initializer"
Jul 28 18:30:52.819779 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819452    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kube-rbac-proxy"
Jul 28 18:30:52.819779 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819463    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="325e2418-1a48-4829-b8be-bb3d4a9b90f3" containerName="kserve-container"
Jul 28 18:30:52.819779 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819473    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="kube-rbac-proxy"
Jul 28 18:30:52.819779 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.819479    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4da4a41-6e26-4a25-8598-60a55743396c" containerName="kserve-container"
Jul 28 18:30:52.822661 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.822642    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.825349 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.825328    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 18:30:52.825540 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.825327    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jul 28 18:30:52.825540 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.825334    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:30:52.825677 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.825611    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jul 28 18:30:52.826439 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.826412    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:30:52.826570 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.826455    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:30:52.826683 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.826658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:30:52.832704 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.832683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"]
Jul 28 18:30:52.860701 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.860673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.860806 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.860713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/146f64cf-4668-42b9-8812-6afee5f0d818-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.860806 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.860733    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.860806 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.860783    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/146f64cf-4668-42b9-8812-6afee5f0d818-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.860958 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.860856    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khlsh\" (UniqueName: \"kubernetes.io/projected/146f64cf-4668-42b9-8812-6afee5f0d818-kube-api-access-khlsh\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.961945 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.961918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khlsh\" (UniqueName: \"kubernetes.io/projected/146f64cf-4668-42b9-8812-6afee5f0d818-kube-api-access-khlsh\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.962054 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.961967    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.962054 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.962017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/146f64cf-4668-42b9-8812-6afee5f0d818-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.962054 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.962042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.962222 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.962071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/146f64cf-4668-42b9-8812-6afee5f0d818-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.962222 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:30:52.962194    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-serving-cert: secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 18:30:52.962363 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:30:52.962289    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146f64cf-4668-42b9-8812-6afee5f0d818-proxy-tls podName:146f64cf-4668-42b9-8812-6afee5f0d818 nodeName:}" failed. No retries permitted until 2026-07-28 18:30:53.462268039 +0000 UTC m=+4812.106237521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/146f64cf-4668-42b9-8812-6afee5f0d818-proxy-tls") pod "isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" (UID: "146f64cf-4668-42b9-8812-6afee5f0d818") : secret "isvc-xgboost-v2-runtime-predictor-serving-cert" not found
Jul 28 18:30:52.962535 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.962515    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/146f64cf-4668-42b9-8812-6afee5f0d818-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.962921 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.962901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.962962 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.962901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:52.970950 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:52.970927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khlsh\" (UniqueName: \"kubernetes.io/projected/146f64cf-4668-42b9-8812-6afee5f0d818-kube-api-access-khlsh\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:53.465200 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:53.465160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/146f64cf-4668-42b9-8812-6afee5f0d818-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:53.467449 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:53.467423    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/146f64cf-4668-42b9-8812-6afee5f0d818-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:53.733685 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:53.733589    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:53.895640 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:53.895610    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"]
Jul 28 18:30:53.897514 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:30:53.897490    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146f64cf_4668_42b9_8812_6afee5f0d818.slice/crio-efda9843d93e40d04e8ed93603e39fe8ab0d0d609deb9e534f2d319d90523ad7 WatchSource:0}: Error finding container efda9843d93e40d04e8ed93603e39fe8ab0d0d609deb9e534f2d319d90523ad7: Status 404 returned error can't find the container with id efda9843d93e40d04e8ed93603e39fe8ab0d0d609deb9e534f2d319d90523ad7
Jul 28 18:30:54.570701 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:54.570664    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" event={"ID":"146f64cf-4668-42b9-8812-6afee5f0d818","Type":"ContainerStarted","Data":"0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e"}
Jul 28 18:30:54.570905 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:54.570727    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" event={"ID":"146f64cf-4668-42b9-8812-6afee5f0d818","Type":"ContainerStarted","Data":"efda9843d93e40d04e8ed93603e39fe8ab0d0d609deb9e534f2d319d90523ad7"}
Jul 28 18:30:57.582810 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:57.582776    2577 generic.go:356] "Generic (PLEG): container finished" podID="146f64cf-4668-42b9-8812-6afee5f0d818" containerID="0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e" exitCode=0
Jul 28 18:30:57.583124 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:57.582840    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" event={"ID":"146f64cf-4668-42b9-8812-6afee5f0d818","Type":"ContainerDied","Data":"0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e"}
Jul 28 18:30:58.589200 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:58.589168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" event={"ID":"146f64cf-4668-42b9-8812-6afee5f0d818","Type":"ContainerStarted","Data":"c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b"}
Jul 28 18:30:58.589607 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:58.589226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" event={"ID":"146f64cf-4668-42b9-8812-6afee5f0d818","Type":"ContainerStarted","Data":"c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d"}
Jul 28 18:30:58.589607 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:58.589577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:58.589691 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:58.589627    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:30:58.614757 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:30:58.614703    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" podStartSLOduration=3.083180266 podStartE2EDuration="6.614688973s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.531508707s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:58.613116361 +0000 UTC m=+4817.257085851" watchObservedRunningTime="2026-07-28 18:30:58.614688973 +0000 UTC m=+4817.258658463"
Jul 28 18:31:04.598584 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:04.598552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:31:16.130188 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:16.130154    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:31:16.136868 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:16.136845    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:31:34.634707 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:34.634612    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jul 28 18:31:44.601041 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:44.601005    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:31:52.889323 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:52.889291    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"]
Jul 28 18:31:52.889823 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:52.889568    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kserve-container" containerID="cri-o://c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d" gracePeriod=30
Jul 28 18:31:52.889823 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:52.889664    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kube-rbac-proxy" containerID="cri-o://c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b" gracePeriod=30
Jul 28 18:31:53.815061 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:53.815028    2577 generic.go:356] "Generic (PLEG): container finished" podID="146f64cf-4668-42b9-8812-6afee5f0d818" containerID="c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b" exitCode=2
Jul 28 18:31:53.815219 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:53.815102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" event={"ID":"146f64cf-4668-42b9-8812-6afee5f0d818","Type":"ContainerDied","Data":"c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b"}
Jul 28 18:31:54.593837 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:54.593794    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 28 18:31:55.640437 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:55.640395    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.44:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 18:31:59.635434 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.635410    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:31:59.713705 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.713637    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/146f64cf-4668-42b9-8812-6afee5f0d818-proxy-tls\") pod \"146f64cf-4668-42b9-8812-6afee5f0d818\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") "
Jul 28 18:31:59.713705 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.713667    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-cabundle-cert\") pod \"146f64cf-4668-42b9-8812-6afee5f0d818\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") "
Jul 28 18:31:59.713705 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.713703    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khlsh\" (UniqueName: \"kubernetes.io/projected/146f64cf-4668-42b9-8812-6afee5f0d818-kube-api-access-khlsh\") pod \"146f64cf-4668-42b9-8812-6afee5f0d818\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") "
Jul 28 18:31:59.713953 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.713739    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-kube-rbac-proxy-sar-config\") pod \"146f64cf-4668-42b9-8812-6afee5f0d818\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") "
Jul 28 18:31:59.713953 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.713761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/146f64cf-4668-42b9-8812-6afee5f0d818-kserve-provision-location\") pod \"146f64cf-4668-42b9-8812-6afee5f0d818\" (UID: \"146f64cf-4668-42b9-8812-6afee5f0d818\") "
Jul 28 18:31:59.714111 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.714082    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "146f64cf-4668-42b9-8812-6afee5f0d818" (UID: "146f64cf-4668-42b9-8812-6afee5f0d818"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:59.714181 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.714103    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "146f64cf-4668-42b9-8812-6afee5f0d818" (UID: "146f64cf-4668-42b9-8812-6afee5f0d818"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:31:59.714181 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.714161    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146f64cf-4668-42b9-8812-6afee5f0d818-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "146f64cf-4668-42b9-8812-6afee5f0d818" (UID: "146f64cf-4668-42b9-8812-6afee5f0d818"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:31:59.715734 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.715697    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146f64cf-4668-42b9-8812-6afee5f0d818-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "146f64cf-4668-42b9-8812-6afee5f0d818" (UID: "146f64cf-4668-42b9-8812-6afee5f0d818"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:31:59.715828 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.715781    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146f64cf-4668-42b9-8812-6afee5f0d818-kube-api-access-khlsh" (OuterVolumeSpecName: "kube-api-access-khlsh") pod "146f64cf-4668-42b9-8812-6afee5f0d818" (UID: "146f64cf-4668-42b9-8812-6afee5f0d818"). InnerVolumeSpecName "kube-api-access-khlsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:59.814599 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.814573    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/146f64cf-4668-42b9-8812-6afee5f0d818-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:31:59.814599 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.814599    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:31:59.814773 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.814612    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khlsh\" (UniqueName: \"kubernetes.io/projected/146f64cf-4668-42b9-8812-6afee5f0d818-kube-api-access-khlsh\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:31:59.814773 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.814622    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/146f64cf-4668-42b9-8812-6afee5f0d818-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:31:59.814773 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.814631    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/146f64cf-4668-42b9-8812-6afee5f0d818-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:31:59.842489 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.842463    2577 generic.go:356] "Generic (PLEG): container finished" podID="146f64cf-4668-42b9-8812-6afee5f0d818" containerID="c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d" exitCode=0
Jul 28 18:31:59.842610 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.842522    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"
Jul 28 18:31:59.842610 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.842547    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" event={"ID":"146f64cf-4668-42b9-8812-6afee5f0d818","Type":"ContainerDied","Data":"c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d"}
Jul 28 18:31:59.842702 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.842630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" event={"ID":"146f64cf-4668-42b9-8812-6afee5f0d818","Type":"ContainerDied","Data":"efda9843d93e40d04e8ed93603e39fe8ab0d0d609deb9e534f2d319d90523ad7"}
Jul 28 18:31:59.842702 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.842657    2577 scope.go:117] "RemoveContainer" containerID="c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b"
Jul 28 18:31:59.851645 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.851630    2577 scope.go:117] "RemoveContainer" containerID="c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d"
Jul 28 18:31:59.858885 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.858869    2577 scope.go:117] "RemoveContainer" containerID="0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e"
Jul 28 18:31:59.866211 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.866193    2577 scope.go:117] "RemoveContainer" containerID="c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b"
Jul 28 18:31:59.866472 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:31:59.866451    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b\": container with ID starting with c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b not found: ID does not exist" containerID="c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b"
Jul 28 18:31:59.866543 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.866472    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"]
Jul 28 18:31:59.866543 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.866478    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b"} err="failed to get container status \"c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b\": rpc error: code = NotFound desc = could not find container \"c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b\": container with ID starting with c207de7816174088c0a24adfd132ffc352d0a23578ce80e0960f72d90c0e1d6b not found: ID does not exist"
Jul 28 18:31:59.866543 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.866498    2577 scope.go:117] "RemoveContainer" containerID="c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d"
Jul 28 18:31:59.866753 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:31:59.866720    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d\": container with ID starting with c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d not found: ID does not exist" containerID="c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d"
Jul 28 18:31:59.866807 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.866759    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d"} err="failed to get container status \"c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d\": rpc error: code = NotFound desc = could not find container \"c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d\": container with ID starting with c148721eb3a3b424a429ccb3165fbe0b3e02a9849b48a06f28740c50e78dc15d not found: ID does not exist"
Jul 28 18:31:59.866807 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.866776    2577 scope.go:117] "RemoveContainer" containerID="0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e"
Jul 28 18:31:59.867071 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:31:59.867048    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e\": container with ID starting with 0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e not found: ID does not exist" containerID="0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e"
Jul 28 18:31:59.867188 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.867083    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e"} err="failed to get container status \"0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e\": rpc error: code = NotFound desc = could not find container \"0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e\": container with ID starting with 0f531f277b06fb87a41e00f219991beca838cf0132d6cc327f9551aa0180731e not found: ID does not exist"
Jul 28 18:31:59.869537 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.869517    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc"]
Jul 28 18:31:59.904075 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:31:59.904054    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" path="/var/lib/kubelet/pods/146f64cf-4668-42b9-8812-6afee5f0d818/volumes"
Jul 28 18:32:00.593945 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:32:00.593905    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-nxhlc" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": context deadline exceeded"
Jul 28 18:34:23.642118 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.642084    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"]
Jul 28 18:34:23.642605 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.642417    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kube-rbac-proxy"
Jul 28 18:34:23.642605 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.642431    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kube-rbac-proxy"
Jul 28 18:34:23.642605 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.642441    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kserve-container"
Jul 28 18:34:23.642605 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.642447    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kserve-container"
Jul 28 18:34:23.642605 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.642461    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="storage-initializer"
Jul 28 18:34:23.642605 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.642467    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="storage-initializer"
Jul 28 18:34:23.642605 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.642510    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kserve-container"
Jul 28 18:34:23.642605 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.642521    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="146f64cf-4668-42b9-8812-6afee5f0d818" containerName="kube-rbac-proxy"
Jul 28 18:34:23.645647 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.645631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.648411 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.648384    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jul 28 18:34:23.648551 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.648393    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jul 28 18:34:23.648551 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.648424    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:34:23.648668 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.648623    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:34:23.649590 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.649570    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:34:23.649711 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.649619    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:34:23.649711 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.649623    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:34:23.655725 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.655700    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"]
Jul 28 18:34:23.708962 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.708933    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27805ac6-8072-4d94-a24b-9aa6c010fae4-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.709071 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.708980    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.709071 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.709027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5dbq\" (UniqueName: \"kubernetes.io/projected/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-api-access-q5dbq\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.709152 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.709076    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27805ac6-8072-4d94-a24b-9aa6c010fae4-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.709152 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.709120    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.810025 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.809995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27805ac6-8072-4d94-a24b-9aa6c010fae4-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.810172 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.810037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.810172 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.810108    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5dbq\" (UniqueName: \"kubernetes.io/projected/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-api-access-q5dbq\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.810309 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.810180    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27805ac6-8072-4d94-a24b-9aa6c010fae4-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.810309 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:34:23.810211    2577 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 18:34:23.810309 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.810229    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.810309 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:34:23.810287    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27805ac6-8072-4d94-a24b-9aa6c010fae4-proxy-tls podName:27805ac6-8072-4d94-a24b-9aa6c010fae4 nodeName:}" failed. No retries permitted until 2026-07-28 18:34:24.310270658 +0000 UTC m=+5022.954240126 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/27805ac6-8072-4d94-a24b-9aa6c010fae4-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" (UID: "27805ac6-8072-4d94-a24b-9aa6c010fae4") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jul 28 18:34:23.810604 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.810586    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27805ac6-8072-4d94-a24b-9aa6c010fae4-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.810971 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.810952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.811081 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.811063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:23.821162 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:23.821135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5dbq\" (UniqueName: \"kubernetes.io/projected/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-api-access-q5dbq\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:24.314570 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:24.314535    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27805ac6-8072-4d94-a24b-9aa6c010fae4-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:24.316865 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:24.316834    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27805ac6-8072-4d94-a24b-9aa6c010fae4-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:24.558344 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:24.558316    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:24.743179 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:24.743154    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"]
Jul 28 18:34:24.745071 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:34:24.745044    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27805ac6_8072_4d94_a24b_9aa6c010fae4.slice/crio-5374fe373f661ce8aa7089d96f0160373909f4a59f0e76390383bc85457640a4 WatchSource:0}: Error finding container 5374fe373f661ce8aa7089d96f0160373909f4a59f0e76390383bc85457640a4: Status 404 returned error can't find the container with id 5374fe373f661ce8aa7089d96f0160373909f4a59f0e76390383bc85457640a4
Jul 28 18:34:25.406564 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:25.406524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" event={"ID":"27805ac6-8072-4d94-a24b-9aa6c010fae4","Type":"ContainerStarted","Data":"2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461"}
Jul 28 18:34:25.406751 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:25.406598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" event={"ID":"27805ac6-8072-4d94-a24b-9aa6c010fae4","Type":"ContainerStarted","Data":"5374fe373f661ce8aa7089d96f0160373909f4a59f0e76390383bc85457640a4"}
Jul 28 18:34:26.411507 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:26.411473    2577 generic.go:356] "Generic (PLEG): container finished" podID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerID="2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461" exitCode=0
Jul 28 18:34:26.411938 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:26.411559    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" event={"ID":"27805ac6-8072-4d94-a24b-9aa6c010fae4","Type":"ContainerDied","Data":"2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461"}
Jul 28 18:34:27.417235 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:27.417197    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" event={"ID":"27805ac6-8072-4d94-a24b-9aa6c010fae4","Type":"ContainerStarted","Data":"7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2"}
Jul 28 18:34:27.417632 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:27.417271    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" event={"ID":"27805ac6-8072-4d94-a24b-9aa6c010fae4","Type":"ContainerStarted","Data":"0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83"}
Jul 28 18:34:27.417632 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:27.417360    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:27.437497 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:27.437452    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podStartSLOduration=3.323605778 podStartE2EDuration="4.437438351s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.113832573s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:27.435336541 +0000 UTC m=+5026.079306035" watchObservedRunningTime="2026-07-28 18:34:27.437438351 +0000 UTC m=+5026.081407840"
Jul 28 18:34:28.421845 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:28.421811    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:28.423026 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:28.422994    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:34:29.425623 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:29.425580    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:34:34.430023 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:34.429945    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:34:34.430624 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:34.430593    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:34:44.430944 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:44.430902    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:34:54.430689 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:34:54.430647    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:35:04.430768 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:04.430722    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:35:14.431069 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:14.431031    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:35:24.430596 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:24.430558    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:35:34.431146 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:34.431107    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused"
Jul 28 18:35:39.904554 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:39.904526    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:35:43.711669 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:43.711638    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"]
Jul 28 18:35:43.712105 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:43.711850    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container" containerID="cri-o://0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83" gracePeriod=30
Jul 28 18:35:43.712105 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:43.711919    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kube-rbac-proxy" containerID="cri-o://7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2" gracePeriod=30
Jul 28 18:35:44.426424 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:44.426386    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.45:8643/healthz\": dial tcp 10.133.0.45:8643: connect: connection refused"
Jul 28 18:35:44.729632 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:44.729548    2577 generic.go:356] "Generic (PLEG): container finished" podID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerID="7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2" exitCode=2
Jul 28 18:35:44.729967 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:44.729620    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" event={"ID":"27805ac6-8072-4d94-a24b-9aa6c010fae4","Type":"ContainerDied","Data":"7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2"}
Jul 28 18:35:47.999451 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:47.999394    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:35:48.060879 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.060849    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27805ac6-8072-4d94-a24b-9aa6c010fae4-proxy-tls\") pod \"27805ac6-8072-4d94-a24b-9aa6c010fae4\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") "
Jul 28 18:35:48.061040 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.060939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-rbac-proxy-sar-config\") pod \"27805ac6-8072-4d94-a24b-9aa6c010fae4\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") "
Jul 28 18:35:48.061040 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.060998    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-cabundle-cert\") pod \"27805ac6-8072-4d94-a24b-9aa6c010fae4\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") "
Jul 28 18:35:48.061040 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.061030    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5dbq\" (UniqueName: \"kubernetes.io/projected/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-api-access-q5dbq\") pod \"27805ac6-8072-4d94-a24b-9aa6c010fae4\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") "
Jul 28 18:35:48.061225 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.061065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27805ac6-8072-4d94-a24b-9aa6c010fae4-kserve-provision-location\") pod \"27805ac6-8072-4d94-a24b-9aa6c010fae4\" (UID: \"27805ac6-8072-4d94-a24b-9aa6c010fae4\") "
Jul 28 18:35:48.061393 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.061325    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27805ac6-8072-4d94-a24b-9aa6c010fae4" (UID: "27805ac6-8072-4d94-a24b-9aa6c010fae4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:35:48.061452 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.061435    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27805ac6-8072-4d94-a24b-9aa6c010fae4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27805ac6-8072-4d94-a24b-9aa6c010fae4" (UID: "27805ac6-8072-4d94-a24b-9aa6c010fae4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:35:48.061733 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.061708    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27805ac6-8072-4d94-a24b-9aa6c010fae4" (UID: "27805ac6-8072-4d94-a24b-9aa6c010fae4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:35:48.063047 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.063018    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-api-access-q5dbq" (OuterVolumeSpecName: "kube-api-access-q5dbq") pod "27805ac6-8072-4d94-a24b-9aa6c010fae4" (UID: "27805ac6-8072-4d94-a24b-9aa6c010fae4"). InnerVolumeSpecName "kube-api-access-q5dbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:35:48.063141 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.063098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27805ac6-8072-4d94-a24b-9aa6c010fae4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27805ac6-8072-4d94-a24b-9aa6c010fae4" (UID: "27805ac6-8072-4d94-a24b-9aa6c010fae4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:35:48.162588 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.162559    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5dbq\" (UniqueName: \"kubernetes.io/projected/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-api-access-q5dbq\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:35:48.162588 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.162583    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27805ac6-8072-4d94-a24b-9aa6c010fae4-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:35:48.162763 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.162595    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27805ac6-8072-4d94-a24b-9aa6c010fae4-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:35:48.162763 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.162607    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:35:48.162763 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.162615    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27805ac6-8072-4d94-a24b-9aa6c010fae4-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:35:48.747686 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.747656    2577 generic.go:356] "Generic (PLEG): container finished" podID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerID="0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83" exitCode=0
Jul 28 18:35:48.747867 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.747712    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"
Jul 28 18:35:48.747867 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.747721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" event={"ID":"27805ac6-8072-4d94-a24b-9aa6c010fae4","Type":"ContainerDied","Data":"0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83"}
Jul 28 18:35:48.747867 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.747789    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf" event={"ID":"27805ac6-8072-4d94-a24b-9aa6c010fae4","Type":"ContainerDied","Data":"5374fe373f661ce8aa7089d96f0160373909f4a59f0e76390383bc85457640a4"}
Jul 28 18:35:48.747867 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.747841    2577 scope.go:117] "RemoveContainer" containerID="7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2"
Jul 28 18:35:48.756432 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.756119    2577 scope.go:117] "RemoveContainer" containerID="0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83"
Jul 28 18:35:48.763329 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.763312    2577 scope.go:117] "RemoveContainer" containerID="2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461"
Jul 28 18:35:48.770937 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.770914    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"]
Jul 28 18:35:48.771381 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.771282    2577 scope.go:117] "RemoveContainer" containerID="7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2"
Jul 28 18:35:48.771663 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:35:48.771612    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2\": container with ID starting with 7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2 not found: ID does not exist" containerID="7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2"
Jul 28 18:35:48.771841 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.771742    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2"} err="failed to get container status \"7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2\": rpc error: code = NotFound desc = could not find container \"7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2\": container with ID starting with 7c43cf06a4d651b2366ec7afaca21701c7ce0983c206fab55e23d4b34f938ba2 not found: ID does not exist"
Jul 28 18:35:48.771841 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.771823    2577 scope.go:117] "RemoveContainer" containerID="0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83"
Jul 28 18:35:48.772162 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:35:48.772137    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83\": container with ID starting with 0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83 not found: ID does not exist" containerID="0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83"
Jul 28 18:35:48.772261 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.772171    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83"} err="failed to get container status \"0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83\": rpc error: code = NotFound desc = could not find container \"0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83\": container with ID starting with 0d1fbcdab512b84b0ec5721b780ef7540d6acb796ce853a6263172f878786c83 not found: ID does not exist"
Jul 28 18:35:48.772261 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.772191    2577 scope.go:117] "RemoveContainer" containerID="2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461"
Jul 28 18:35:48.772491 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:35:48.772470    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461\": container with ID starting with 2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461 not found: ID does not exist" containerID="2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461"
Jul 28 18:35:48.772544 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.772497    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461"} err="failed to get container status \"2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461\": rpc error: code = NotFound desc = could not find container \"2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461\": container with ID starting with 2ed125356def83f3e35f0be5a5b902add32ebd74c0426401a9016d10580ca461 not found: ID does not exist"
Jul 28 18:35:48.773772 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:48.773757    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-686f497c9c-5z4kf"]
Jul 28 18:35:49.904389 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:49.904356    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" path="/var/lib/kubelet/pods/27805ac6-8072-4d94-a24b-9aa6c010fae4/volumes"
Jul 28 18:35:55.895793 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.895759    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"]
Jul 28 18:35:55.896150 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.896080    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container"
Jul 28 18:35:55.896150 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.896093    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container"
Jul 28 18:35:55.896150 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.896102    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kube-rbac-proxy"
Jul 28 18:35:55.896150 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.896107    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kube-rbac-proxy"
Jul 28 18:35:55.896150 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.896117    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="storage-initializer"
Jul 28 18:35:55.896150 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.896123    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="storage-initializer"
Jul 28 18:35:55.896379 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.896172    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kube-rbac-proxy"
Jul 28 18:35:55.896379 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.896183    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="27805ac6-8072-4d94-a24b-9aa6c010fae4" containerName="kserve-container"
Jul 28 18:35:55.900876 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.900847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:55.903644 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.903625    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:35:55.903809 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.903662    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jul 28 18:35:55.903894 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.903880    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:35:55.905054 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.904933    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:35:55.905054 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.904955    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:35:55.905054 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.904972    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jul 28 18:35:55.905054 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.904959    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:35:55.910665 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:55.910644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"]
Jul 28 18:35:56.024934 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.024904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.025073 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.024941    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.025073 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.024966    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.025073 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.025031    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tdnv\" (UniqueName: \"kubernetes.io/projected/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-api-access-6tdnv\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.025187 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.025144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.126402 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.126370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.126566 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.126409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.126566 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.126434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.126566 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.126460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tdnv\" (UniqueName: \"kubernetes.io/projected/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-api-access-6tdnv\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.126566 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.126536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.126971 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.126945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.127203 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.127182    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.127321 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.127301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.129053 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.129035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.134447 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.134427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tdnv\" (UniqueName: \"kubernetes.io/projected/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-api-access-6tdnv\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.212655 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.212554    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:56.338155 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.338128    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"]
Jul 28 18:35:56.340192 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:35:56.340170    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7a6ff0_e7f3_4472_8c3b_3d545e70245a.slice/crio-8197bfbc710623bd0f3086ce99f190b9bec177227d6ec8edfd06995483f1aff6 WatchSource:0}: Error finding container 8197bfbc710623bd0f3086ce99f190b9bec177227d6ec8edfd06995483f1aff6: Status 404 returned error can't find the container with id 8197bfbc710623bd0f3086ce99f190b9bec177227d6ec8edfd06995483f1aff6
Jul 28 18:35:56.782236 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.782195    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" event={"ID":"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a","Type":"ContainerStarted","Data":"0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6"}
Jul 28 18:35:56.782466 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:56.782321    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" event={"ID":"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a","Type":"ContainerStarted","Data":"8197bfbc710623bd0f3086ce99f190b9bec177227d6ec8edfd06995483f1aff6"}
Jul 28 18:35:57.786861 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:57.786774    2577 generic.go:356] "Generic (PLEG): container finished" podID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerID="0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6" exitCode=0
Jul 28 18:35:57.787286 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:57.786863    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" event={"ID":"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a","Type":"ContainerDied","Data":"0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6"}
Jul 28 18:35:58.793807 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:58.793773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" event={"ID":"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a","Type":"ContainerStarted","Data":"293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b"}
Jul 28 18:35:58.794311 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:58.793842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" event={"ID":"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a","Type":"ContainerStarted","Data":"d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d"}
Jul 28 18:35:58.794311 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:58.793876    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:58.794311 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:58.793906    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:35:58.795229 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:58.795200    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:35:58.815616 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:58.815571    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podStartSLOduration=2.787262097 podStartE2EDuration="3.815556669s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.028294572s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:35:58.814603106 +0000 UTC m=+5117.458572597" watchObservedRunningTime="2026-07-28 18:35:58.815556669 +0000 UTC m=+5117.459526159"
Jul 28 18:35:59.802440 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:35:59.802391    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:36:04.806918 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:36:04.806844    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:36:04.807345 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:36:04.807320    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:36:14.807832 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:36:14.807787    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:36:16.162013 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:36:16.161986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:36:16.164156 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:36:16.164132    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:36:24.807670 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:36:24.807631    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:36:34.807499 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:36:34.807456    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:36:44.807338 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:36:44.807289    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:36:54.807547 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:36:54.807508    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:37:04.807564 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:04.807526    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused"
Jul 28 18:37:14.808447 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:14.808406    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:37:15.935185 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:15.935156    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"]
Jul 28 18:37:15.935611 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:15.935392    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container" containerID="cri-o://d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d" gracePeriod=30
Jul 28 18:37:15.935611 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:15.935429    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kube-rbac-proxy" containerID="cri-o://293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b" gracePeriod=30
Jul 28 18:37:16.108183 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:16.108147    2577 generic.go:356] "Generic (PLEG): container finished" podID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerID="293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b" exitCode=2
Jul 28 18:37:16.108364 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:16.108230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" event={"ID":"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a","Type":"ContainerDied","Data":"293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b"}
Jul 28 18:37:19.803080 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:19.803043    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.46:8643/healthz\": dial tcp 10.133.0.46:8643: connect: connection refused"
Jul 28 18:37:20.491364 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.491340    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:37:20.619194 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.619107    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-cabundle-cert\") pod \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") "
Jul 28 18:37:20.619194 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.619173    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kserve-provision-location\") pod \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") "
Jul 28 18:37:20.619194 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.619194    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-rbac-proxy-sar-config\") pod \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") "
Jul 28 18:37:20.619484 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.619212    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tdnv\" (UniqueName: \"kubernetes.io/projected/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-api-access-6tdnv\") pod \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") "
Jul 28 18:37:20.619484 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.619260    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-proxy-tls\") pod \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\" (UID: \"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a\") "
Jul 28 18:37:20.619601 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.619570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" (UID: "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:37:20.619601 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.619570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" (UID: "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:37:20.619715 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.619575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" (UID: "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:37:20.621357 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.621332    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" (UID: "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:37:20.621484 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.621465    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-api-access-6tdnv" (OuterVolumeSpecName: "kube-api-access-6tdnv") pod "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" (UID: "9b7a6ff0-e7f3-4472-8c3b-3d545e70245a"). InnerVolumeSpecName "kube-api-access-6tdnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:37:20.720599 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.720549    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:37:20.720599 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.720592    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:37:20.720599 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.720606    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:37:20.720844 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.720623    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:37:20.720844 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:20.720635    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tdnv\" (UniqueName: \"kubernetes.io/projected/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a-kube-api-access-6tdnv\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:37:21.129664 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.129628    2577 generic.go:356] "Generic (PLEG): container finished" podID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerID="d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d" exitCode=0
Jul 28 18:37:21.130135 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.129686    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"
Jul 28 18:37:21.130135 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.129716    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" event={"ID":"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a","Type":"ContainerDied","Data":"d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d"}
Jul 28 18:37:21.130135 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.129792    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj" event={"ID":"9b7a6ff0-e7f3-4472-8c3b-3d545e70245a","Type":"ContainerDied","Data":"8197bfbc710623bd0f3086ce99f190b9bec177227d6ec8edfd06995483f1aff6"}
Jul 28 18:37:21.130135 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.129834    2577 scope.go:117] "RemoveContainer" containerID="293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b"
Jul 28 18:37:21.142301 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.139595    2577 scope.go:117] "RemoveContainer" containerID="d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d"
Jul 28 18:37:21.147894 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.147872    2577 scope.go:117] "RemoveContainer" containerID="0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6"
Jul 28 18:37:21.155380 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.155359    2577 scope.go:117] "RemoveContainer" containerID="293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b"
Jul 28 18:37:21.155644 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:37:21.155626    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b\": container with ID starting with 293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b not found: ID does not exist" containerID="293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b"
Jul 28 18:37:21.155697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.155653    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b"} err="failed to get container status \"293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b\": rpc error: code = NotFound desc = could not find container \"293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b\": container with ID starting with 293b5ca58ac80cf2f1e14b434eacaae139761e41440d36b46d5236a6a4023f0b not found: ID does not exist"
Jul 28 18:37:21.155697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.155671    2577 scope.go:117] "RemoveContainer" containerID="d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d"
Jul 28 18:37:21.155900 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:37:21.155886    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d\": container with ID starting with d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d not found: ID does not exist" containerID="d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d"
Jul 28 18:37:21.155939 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.155903    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d"} err="failed to get container status \"d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d\": rpc error: code = NotFound desc = could not find container \"d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d\": container with ID starting with d8c7abc99818d5a8c6f76d699dd728d779767d7d4bb70cf65f22eb5fbbb0c33d not found: ID does not exist"
Jul 28 18:37:21.155939 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.155915    2577 scope.go:117] "RemoveContainer" containerID="0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6"
Jul 28 18:37:21.156084 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:37:21.156071    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6\": container with ID starting with 0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6 not found: ID does not exist" containerID="0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6"
Jul 28 18:37:21.156128 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.156085    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6"} err="failed to get container status \"0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6\": rpc error: code = NotFound desc = could not find container \"0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6\": container with ID starting with 0a5a9127dd303eb2fef8cc06827be023b50731068b1523587f55d19f4324b9b6 not found: ID does not exist"
Jul 28 18:37:21.156359 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.156341    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"]
Jul 28 18:37:21.159567 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.159548    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-6449bd88bf-rgrkj"]
Jul 28 18:37:21.904081 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:21.904044    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" path="/var/lib/kubelet/pods/9b7a6ff0-e7f3-4472-8c3b-3d545e70245a/volumes"
Jul 28 18:37:28.127775 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.127736    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"]
Jul 28 18:37:28.128328 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.128179    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="storage-initializer"
Jul 28 18:37:28.128328 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.128199    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="storage-initializer"
Jul 28 18:37:28.128328 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.128233    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container"
Jul 28 18:37:28.128328 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.128259    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container"
Jul 28 18:37:28.128328 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.128297    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kube-rbac-proxy"
Jul 28 18:37:28.128328 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.128306    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kube-rbac-proxy"
Jul 28 18:37:28.128659 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.128383    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kserve-container"
Jul 28 18:37:28.128659 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.128396    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b7a6ff0-e7f3-4472-8c3b-3d545e70245a" containerName="kube-rbac-proxy"
Jul 28 18:37:28.133305 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.133285    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.136525 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.136501    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 18:37:28.136663 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.136640    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jul 28 18:37:28.136741 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.136648    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 28 18:37:28.136741 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.136670    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 18:37:28.137725 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.137698    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 18:37:28.137832 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.137699    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dn2w2\""
Jul 28 18:37:28.137832 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.137703    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jul 28 18:37:28.144072 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.144050    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"]
Jul 28 18:37:28.285544 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.285504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8ttn\" (UniqueName: \"kubernetes.io/projected/91510293-b028-483a-9c1e-bf34e17caaa2-kube-api-access-l8ttn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.285544 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.285544    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91510293-b028-483a-9c1e-bf34e17caaa2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.285768 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.285604    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.285768 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.285642    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91510293-b028-483a-9c1e-bf34e17caaa2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.285768 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.285687    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.386207 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.386177    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.386365 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.386241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8ttn\" (UniqueName: \"kubernetes.io/projected/91510293-b028-483a-9c1e-bf34e17caaa2-kube-api-access-l8ttn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.386365 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.386280    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91510293-b028-483a-9c1e-bf34e17caaa2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.386365 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.386307    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.386365 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.386325    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91510293-b028-483a-9c1e-bf34e17caaa2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.386685 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.386656    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91510293-b028-483a-9c1e-bf34e17caaa2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.386951 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.386929    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.387034 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.386968    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.388659 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.388640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91510293-b028-483a-9c1e-bf34e17caaa2-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.395406 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.395385    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8ttn\" (UniqueName: \"kubernetes.io/projected/91510293-b028-483a-9c1e-bf34e17caaa2-kube-api-access-l8ttn\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.444953 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.444927    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:28.572019 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:28.571996    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"]
Jul 28 18:37:28.574222 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:37:28.574193    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91510293_b028_483a_9c1e_bf34e17caaa2.slice/crio-9ba3ea0c4da542684de7cb270eac2e3fc7e9431c3bd08fbf490353b3eae4d3d7 WatchSource:0}: Error finding container 9ba3ea0c4da542684de7cb270eac2e3fc7e9431c3bd08fbf490353b3eae4d3d7: Status 404 returned error can't find the container with id 9ba3ea0c4da542684de7cb270eac2e3fc7e9431c3bd08fbf490353b3eae4d3d7
Jul 28 18:37:29.170915 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:29.170878    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" event={"ID":"91510293-b028-483a-9c1e-bf34e17caaa2","Type":"ContainerStarted","Data":"5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b"}
Jul 28 18:37:29.171315 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:29.170942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" event={"ID":"91510293-b028-483a-9c1e-bf34e17caaa2","Type":"ContainerStarted","Data":"9ba3ea0c4da542684de7cb270eac2e3fc7e9431c3bd08fbf490353b3eae4d3d7"}
Jul 28 18:37:30.176597 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:30.176565    2577 generic.go:356] "Generic (PLEG): container finished" podID="91510293-b028-483a-9c1e-bf34e17caaa2" containerID="5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b" exitCode=0
Jul 28 18:37:30.176988 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:30.176612    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" event={"ID":"91510293-b028-483a-9c1e-bf34e17caaa2","Type":"ContainerDied","Data":"5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b"}
Jul 28 18:37:31.183278 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:31.183226    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" event={"ID":"91510293-b028-483a-9c1e-bf34e17caaa2","Type":"ContainerStarted","Data":"91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863"}
Jul 28 18:37:31.183672 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:31.183302    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" event={"ID":"91510293-b028-483a-9c1e-bf34e17caaa2","Type":"ContainerStarted","Data":"c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890"}
Jul 28 18:37:31.183672 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:31.183348    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:31.204735 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:31.204683    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podStartSLOduration=2.196603242 podStartE2EDuration="3.204668976s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.008065734s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:37:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:37:31.203134137 +0000 UTC m=+5209.847103627" watchObservedRunningTime="2026-07-28 18:37:31.204668976 +0000 UTC m=+5209.848638465"
Jul 28 18:37:32.192266 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:32.190834    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:32.192759 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:32.192718    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:37:33.191826 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:33.191783    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:37:38.197325 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:38.197295    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:37:38.197918 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:38.197893    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:37:48.198558 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:48.198517    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:37:58.198674 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:37:58.198635    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:38:08.197846 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:08.197801    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:38:18.197944 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:18.197900    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:38:28.198207 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:28.198165    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:38:38.197872 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:38.197832    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.47:8080: connect: connection refused"
Jul 28 18:38:48.198401 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:48.198369    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:38:58.219271 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:58.219216    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"]
Jul 28 18:38:58.219684 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:58.219624    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container" containerID="cri-o://c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890" gracePeriod=30
Jul 28 18:38:58.219772 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:58.219707    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kube-rbac-proxy" containerID="cri-o://91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863" gracePeriod=30
Jul 28 18:38:58.546103 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:58.546021    2577 generic.go:356] "Generic (PLEG): container finished" podID="91510293-b028-483a-9c1e-bf34e17caaa2" containerID="91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863" exitCode=2
Jul 28 18:38:58.546255 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:38:58.546094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" event={"ID":"91510293-b028-483a-9c1e-bf34e17caaa2","Type":"ContainerDied","Data":"91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863"}
Jul 28 18:39:02.763661 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.763638    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:39:02.867710 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.867616    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8ttn\" (UniqueName: \"kubernetes.io/projected/91510293-b028-483a-9c1e-bf34e17caaa2-kube-api-access-l8ttn\") pod \"91510293-b028-483a-9c1e-bf34e17caaa2\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") "
Jul 28 18:39:02.867710 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.867655    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-cabundle-cert\") pod \"91510293-b028-483a-9c1e-bf34e17caaa2\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") "
Jul 28 18:39:02.867710 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.867677    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-kube-rbac-proxy-sar-config\") pod \"91510293-b028-483a-9c1e-bf34e17caaa2\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") "
Jul 28 18:39:02.867953 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.867727    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91510293-b028-483a-9c1e-bf34e17caaa2-kserve-provision-location\") pod \"91510293-b028-483a-9c1e-bf34e17caaa2\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") "
Jul 28 18:39:02.867953 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.867762    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91510293-b028-483a-9c1e-bf34e17caaa2-proxy-tls\") pod \"91510293-b028-483a-9c1e-bf34e17caaa2\" (UID: \"91510293-b028-483a-9c1e-bf34e17caaa2\") "
Jul 28 18:39:02.868092 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.868056    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91510293-b028-483a-9c1e-bf34e17caaa2" (UID: "91510293-b028-483a-9c1e-bf34e17caaa2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:39:02.868092 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.868063    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91510293-b028-483a-9c1e-bf34e17caaa2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "91510293-b028-483a-9c1e-bf34e17caaa2" (UID: "91510293-b028-483a-9c1e-bf34e17caaa2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:02.868092 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.868090    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "91510293-b028-483a-9c1e-bf34e17caaa2" (UID: "91510293-b028-483a-9c1e-bf34e17caaa2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:39:02.869779 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.869748    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91510293-b028-483a-9c1e-bf34e17caaa2-kube-api-access-l8ttn" (OuterVolumeSpecName: "kube-api-access-l8ttn") pod "91510293-b028-483a-9c1e-bf34e17caaa2" (UID: "91510293-b028-483a-9c1e-bf34e17caaa2"). InnerVolumeSpecName "kube-api-access-l8ttn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:39:02.869867 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.869836    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91510293-b028-483a-9c1e-bf34e17caaa2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91510293-b028-483a-9c1e-bf34e17caaa2" (UID: "91510293-b028-483a-9c1e-bf34e17caaa2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:39:02.968661 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.968623    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8ttn\" (UniqueName: \"kubernetes.io/projected/91510293-b028-483a-9c1e-bf34e17caaa2-kube-api-access-l8ttn\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:39:02.968661 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.968652    2577 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-cabundle-cert\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:39:02.968661 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.968668    2577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91510293-b028-483a-9c1e-bf34e17caaa2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:39:02.968892 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.968681    2577 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/91510293-b028-483a-9c1e-bf34e17caaa2-kserve-provision-location\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:39:02.968892 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:02.968693    2577 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91510293-b028-483a-9c1e-bf34e17caaa2-proxy-tls\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:39:03.575199 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.575101    2577 generic.go:356] "Generic (PLEG): container finished" podID="91510293-b028-483a-9c1e-bf34e17caaa2" containerID="c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890" exitCode=0
Jul 28 18:39:03.575199 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.575179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" event={"ID":"91510293-b028-483a-9c1e-bf34e17caaa2","Type":"ContainerDied","Data":"c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890"}
Jul 28 18:39:03.575418 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.575194    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"
Jul 28 18:39:03.575418 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.575318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94" event={"ID":"91510293-b028-483a-9c1e-bf34e17caaa2","Type":"ContainerDied","Data":"9ba3ea0c4da542684de7cb270eac2e3fc7e9431c3bd08fbf490353b3eae4d3d7"}
Jul 28 18:39:03.575418 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.575365    2577 scope.go:117] "RemoveContainer" containerID="91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863"
Jul 28 18:39:03.584939 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.584922    2577 scope.go:117] "RemoveContainer" containerID="c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890"
Jul 28 18:39:03.592166 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.592148    2577 scope.go:117] "RemoveContainer" containerID="5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b"
Jul 28 18:39:03.598209 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.598184    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"]
Jul 28 18:39:03.599582 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.599564    2577 scope.go:117] "RemoveContainer" containerID="91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863"
Jul 28 18:39:03.600034 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:03.600013    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863\": container with ID starting with 91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863 not found: ID does not exist" containerID="91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863"
Jul 28 18:39:03.600110 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.600041    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863"} err="failed to get container status \"91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863\": rpc error: code = NotFound desc = could not find container \"91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863\": container with ID starting with 91a476cd8f77732960c01efb679ac9d8fb0b6d18d0ad1b4184dc8320d9ae9863 not found: ID does not exist"
Jul 28 18:39:03.600110 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.600057    2577 scope.go:117] "RemoveContainer" containerID="c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890"
Jul 28 18:39:03.600334 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:03.600316    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890\": container with ID starting with c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890 not found: ID does not exist" containerID="c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890"
Jul 28 18:39:03.600381 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.600340    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890"} err="failed to get container status \"c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890\": rpc error: code = NotFound desc = could not find container \"c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890\": container with ID starting with c89052f51bacd518aa1c0ad468c942eb69ff5f6f78d8bac05b1fc21d8e8a0890 not found: ID does not exist"
Jul 28 18:39:03.600381 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.600354    2577 scope.go:117] "RemoveContainer" containerID="5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b"
Jul 28 18:39:03.600602 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:03.600586    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b\": container with ID starting with 5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b not found: ID does not exist" containerID="5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b"
Jul 28 18:39:03.600673 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.600606    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b"} err="failed to get container status \"5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b\": rpc error: code = NotFound desc = could not find container \"5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b\": container with ID starting with 5a5110a02d3b92581349dfb494d8ebbb9b60b0cd7317af6bf9318bec3a00287b not found: ID does not exist"
Jul 28 18:39:03.601736 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.601720    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-58c5c9b874-mxq94"]
Jul 28 18:39:03.904951 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:03.904910    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" path="/var/lib/kubelet/pods/91510293-b028-483a-9c1e-bf34e17caaa2/volumes"
Jul 28 18:39:11.381158 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.381122    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lw9cf/must-gather-czcp9"]
Jul 28 18:39:11.381655 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.381517    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="storage-initializer"
Jul 28 18:39:11.381655 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.381529    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="storage-initializer"
Jul 28 18:39:11.381655 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.381540    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container"
Jul 28 18:39:11.381655 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.381546    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container"
Jul 28 18:39:11.381655 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.381552    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kube-rbac-proxy"
Jul 28 18:39:11.381655 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.381559    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kube-rbac-proxy"
Jul 28 18:39:11.381655 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.381603    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kserve-container"
Jul 28 18:39:11.381655 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.381611    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="91510293-b028-483a-9c1e-bf34e17caaa2" containerName="kube-rbac-proxy"
Jul 28 18:39:11.384549 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.384533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:11.387192 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.387162    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lw9cf\"/\"kube-root-ca.crt\""
Jul 28 18:39:11.387192 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.387177    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lw9cf\"/\"openshift-service-ca.crt\""
Jul 28 18:39:11.393594 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.393570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lw9cf/must-gather-czcp9"]
Jul 28 18:39:11.538815 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.538771    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59d0eb78-309e-4224-b975-a56d274f2dc6-must-gather-output\") pod \"must-gather-czcp9\" (UID: \"59d0eb78-309e-4224-b975-a56d274f2dc6\") " pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:11.539000 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.538831    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh252\" (UniqueName: \"kubernetes.io/projected/59d0eb78-309e-4224-b975-a56d274f2dc6-kube-api-access-hh252\") pod \"must-gather-czcp9\" (UID: \"59d0eb78-309e-4224-b975-a56d274f2dc6\") " pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:11.639298 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.639269    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh252\" (UniqueName: \"kubernetes.io/projected/59d0eb78-309e-4224-b975-a56d274f2dc6-kube-api-access-hh252\") pod \"must-gather-czcp9\" (UID: \"59d0eb78-309e-4224-b975-a56d274f2dc6\") " pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:11.639466 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.639341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59d0eb78-309e-4224-b975-a56d274f2dc6-must-gather-output\") pod \"must-gather-czcp9\" (UID: \"59d0eb78-309e-4224-b975-a56d274f2dc6\") " pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:11.639632 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.639617    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59d0eb78-309e-4224-b975-a56d274f2dc6-must-gather-output\") pod \"must-gather-czcp9\" (UID: \"59d0eb78-309e-4224-b975-a56d274f2dc6\") " pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:11.648688 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.648650    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh252\" (UniqueName: \"kubernetes.io/projected/59d0eb78-309e-4224-b975-a56d274f2dc6-kube-api-access-hh252\") pod \"must-gather-czcp9\" (UID: \"59d0eb78-309e-4224-b975-a56d274f2dc6\") " pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:11.707901 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.707870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:11.833586 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.833559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lw9cf/must-gather-czcp9"]
Jul 28 18:39:11.836163 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:39:11.836135    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59d0eb78_309e_4224_b975_a56d274f2dc6.slice/crio-6d5a8c1e72dcc9a11b282047c38a6391ce122042a239d7a0cb1ba427694f6c53 WatchSource:0}: Error finding container 6d5a8c1e72dcc9a11b282047c38a6391ce122042a239d7a0cb1ba427694f6c53: Status 404 returned error can't find the container with id 6d5a8c1e72dcc9a11b282047c38a6391ce122042a239d7a0cb1ba427694f6c53
Jul 28 18:39:11.837906 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:11.837890    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:39:12.618512 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:12.618473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lw9cf/must-gather-czcp9" event={"ID":"59d0eb78-309e-4224-b975-a56d274f2dc6","Type":"ContainerStarted","Data":"6d5a8c1e72dcc9a11b282047c38a6391ce122042a239d7a0cb1ba427694f6c53"}
Jul 28 18:39:18.650508 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:18.650470    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lw9cf/must-gather-czcp9" event={"ID":"59d0eb78-309e-4224-b975-a56d274f2dc6","Type":"ContainerStarted","Data":"895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a"}
Jul 28 18:39:18.650922 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:18.650516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lw9cf/must-gather-czcp9" event={"ID":"59d0eb78-309e-4224-b975-a56d274f2dc6","Type":"ContainerStarted","Data":"5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9"}
Jul 28 18:39:18.666815 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:18.666753    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-lw9cf/must-gather-czcp9" podStartSLOduration=1.617490207 podStartE2EDuration="7.666732921s" totalImagesPullingTime="6.049242714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:39:18.666372473 +0000 UTC m=+5317.310341991" watchObservedRunningTime="2026-07-28 18:39:18.666732921 +0000 UTC m=+5317.310702412"
Jul 28 18:39:44.771017 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:44.770985    2577 generic.go:356] "Generic (PLEG): container finished" podID="59d0eb78-309e-4224-b975-a56d274f2dc6" containerID="5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9" exitCode=0
Jul 28 18:39:44.771472 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:44.771055    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lw9cf/must-gather-czcp9" event={"ID":"59d0eb78-309e-4224-b975-a56d274f2dc6","Type":"ContainerDied","Data":"5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9"}
Jul 28 18:39:44.771472 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:44.771360    2577 scope.go:117] "RemoveContainer" containerID="5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9"
Jul 28 18:39:45.251570 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:45.251544    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lw9cf_must-gather-czcp9_59d0eb78-309e-4224-b975-a56d274f2dc6/gather/0.log"
Jul 28 18:39:45.995084 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:45.995050    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kgbts/must-gather-sqjfx"]
Jul 28 18:39:45.998524 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:45.998504    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgbts/must-gather-sqjfx"
Jul 28 18:39:46.001458 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.001437    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kgbts\"/\"openshift-service-ca.crt\""
Jul 28 18:39:46.001572 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.001459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kgbts\"/\"default-dockercfg-rnxxc\""
Jul 28 18:39:46.002680 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.002658    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kgbts\"/\"kube-root-ca.crt\""
Jul 28 18:39:46.006261 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.006222    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgbts/must-gather-sqjfx"]
Jul 28 18:39:46.024817 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.024785    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03c9b7cf-b10c-4249-8089-85872fa71f54-must-gather-output\") pod \"must-gather-sqjfx\" (UID: \"03c9b7cf-b10c-4249-8089-85872fa71f54\") " pod="openshift-must-gather-kgbts/must-gather-sqjfx"
Jul 28 18:39:46.024969 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.024857    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prms9\" (UniqueName: \"kubernetes.io/projected/03c9b7cf-b10c-4249-8089-85872fa71f54-kube-api-access-prms9\") pod \"must-gather-sqjfx\" (UID: \"03c9b7cf-b10c-4249-8089-85872fa71f54\") " pod="openshift-must-gather-kgbts/must-gather-sqjfx"
Jul 28 18:39:46.125769 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.125731    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prms9\" (UniqueName: \"kubernetes.io/projected/03c9b7cf-b10c-4249-8089-85872fa71f54-kube-api-access-prms9\") pod \"must-gather-sqjfx\" (UID: \"03c9b7cf-b10c-4249-8089-85872fa71f54\") " pod="openshift-must-gather-kgbts/must-gather-sqjfx"
Jul 28 18:39:46.125942 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.125783    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03c9b7cf-b10c-4249-8089-85872fa71f54-must-gather-output\") pod \"must-gather-sqjfx\" (UID: \"03c9b7cf-b10c-4249-8089-85872fa71f54\") " pod="openshift-must-gather-kgbts/must-gather-sqjfx"
Jul 28 18:39:46.126169 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.126152    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/03c9b7cf-b10c-4249-8089-85872fa71f54-must-gather-output\") pod \"must-gather-sqjfx\" (UID: \"03c9b7cf-b10c-4249-8089-85872fa71f54\") " pod="openshift-must-gather-kgbts/must-gather-sqjfx"
Jul 28 18:39:46.133592 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.133560    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prms9\" (UniqueName: \"kubernetes.io/projected/03c9b7cf-b10c-4249-8089-85872fa71f54-kube-api-access-prms9\") pod \"must-gather-sqjfx\" (UID: \"03c9b7cf-b10c-4249-8089-85872fa71f54\") " pod="openshift-must-gather-kgbts/must-gather-sqjfx"
Jul 28 18:39:46.307984 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.307907    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgbts/must-gather-sqjfx"
Jul 28 18:39:46.462905 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.462881    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgbts/must-gather-sqjfx"]
Jul 28 18:39:46.464643 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:39:46.464619    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03c9b7cf_b10c_4249_8089_85872fa71f54.slice/crio-0461ba847f0e3fba8f95bc16434bc36a0d81b7e1d26632a53995888534d616e3 WatchSource:0}: Error finding container 0461ba847f0e3fba8f95bc16434bc36a0d81b7e1d26632a53995888534d616e3: Status 404 returned error can't find the container with id 0461ba847f0e3fba8f95bc16434bc36a0d81b7e1d26632a53995888534d616e3
Jul 28 18:39:46.779835 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:46.779797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgbts/must-gather-sqjfx" event={"ID":"03c9b7cf-b10c-4249-8089-85872fa71f54","Type":"ContainerStarted","Data":"0461ba847f0e3fba8f95bc16434bc36a0d81b7e1d26632a53995888534d616e3"}
Jul 28 18:39:47.788480 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:47.788437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgbts/must-gather-sqjfx" event={"ID":"03c9b7cf-b10c-4249-8089-85872fa71f54","Type":"ContainerStarted","Data":"2ecb20ee56be4f05202aa1b1b5daf90263d4ff85fa023d3b637a36820a56e401"}
Jul 28 18:39:47.788983 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:47.788495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgbts/must-gather-sqjfx" event={"ID":"03c9b7cf-b10c-4249-8089-85872fa71f54","Type":"ContainerStarted","Data":"9c99cfe47481851bdd0adef936c4abfab2b985185810923df7674b85a4911ff8"}
Jul 28 18:39:47.806575 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:47.806521    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kgbts/must-gather-sqjfx" podStartSLOduration=1.9358970819999999 podStartE2EDuration="2.806506717s" totalImagesPullingTime="870.609635ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:39:47.80493802 +0000 UTC m=+5346.448907535" watchObservedRunningTime="2026-07-28 18:39:47.806506717 +0000 UTC m=+5346.450476207"
Jul 28 18:39:48.933679 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:48.933646    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c6vbq_4a0a67e1-6d79-423b-8995-cd624597642f/global-pull-secret-syncer/0.log"
Jul 28 18:39:49.097177 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:49.097142    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vvkr9_e2a29023-9a19-4533-afcf-b3ea0ac28e63/konnectivity-agent/0.log"
Jul 28 18:39:49.126227 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:49.126201    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-190.ec2.internal_eee64e253de37dcdd5d1e7fdc42d07fa/haproxy/0.log"
Jul 28 18:39:50.826931 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:50.826879    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-lw9cf/must-gather-czcp9"]
Jul 28 18:39:50.827460 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:50.827182    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-lw9cf/must-gather-czcp9" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" containerName="copy" containerID="cri-o://895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a" gracePeriod=2
Jul 28 18:39:50.831001 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:50.830964    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-lw9cf/must-gather-czcp9"]
Jul 28 18:39:50.831756 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:50.831497    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-czcp9\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lw9cf\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:51.201768 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.201667    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lw9cf_must-gather-czcp9_59d0eb78-309e-4224-b975-a56d274f2dc6/copy/0.log"
Jul 28 18:39:51.205126 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.204876    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:51.207880 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:51.207692    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-czcp9\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lw9cf\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:51.299533 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.298946    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59d0eb78-309e-4224-b975-a56d274f2dc6-must-gather-output\") pod \"59d0eb78-309e-4224-b975-a56d274f2dc6\" (UID: \"59d0eb78-309e-4224-b975-a56d274f2dc6\") "
Jul 28 18:39:51.299533 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.298996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh252\" (UniqueName: \"kubernetes.io/projected/59d0eb78-309e-4224-b975-a56d274f2dc6-kube-api-access-hh252\") pod \"59d0eb78-309e-4224-b975-a56d274f2dc6\" (UID: \"59d0eb78-309e-4224-b975-a56d274f2dc6\") "
Jul 28 18:39:51.302090 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.301151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59d0eb78-309e-4224-b975-a56d274f2dc6-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "59d0eb78-309e-4224-b975-a56d274f2dc6" (UID: "59d0eb78-309e-4224-b975-a56d274f2dc6"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:39:51.310218 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.310181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59d0eb78-309e-4224-b975-a56d274f2dc6-kube-api-access-hh252" (OuterVolumeSpecName: "kube-api-access-hh252") pod "59d0eb78-309e-4224-b975-a56d274f2dc6" (UID: "59d0eb78-309e-4224-b975-a56d274f2dc6"). InnerVolumeSpecName "kube-api-access-hh252". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:39:51.400528 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.400490    2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/59d0eb78-309e-4224-b975-a56d274f2dc6-must-gather-output\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:39:51.400528 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.400529    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh252\" (UniqueName: \"kubernetes.io/projected/59d0eb78-309e-4224-b975-a56d274f2dc6-kube-api-access-hh252\") on node \"ip-10-0-139-190.ec2.internal\" DevicePath \"\""
Jul 28 18:39:51.815817 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.815774    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-lw9cf_must-gather-czcp9_59d0eb78-309e-4224-b975-a56d274f2dc6/copy/0.log"
Jul 28 18:39:51.816353 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.816328    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:51.816473 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.816400    2577 scope.go:117] "RemoveContainer" containerID="895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a"
Jul 28 18:39:51.817297 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.817268    2577 generic.go:356] "Generic (PLEG): container finished" podID="59d0eb78-309e-4224-b975-a56d274f2dc6" containerID="895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a" exitCode=143
Jul 28 18:39:51.819662 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:51.819107    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-czcp9\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lw9cf\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:51.835377 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.834321    2577 scope.go:117] "RemoveContainer" containerID="5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9"
Jul 28 18:39:51.843807 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:51.843477    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-czcp9\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lw9cf\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:51.858422 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.857700    2577 scope.go:117] "RemoveContainer" containerID="895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a"
Jul 28 18:39:51.858422 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:51.858139    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a\": container with ID starting with 895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a not found: ID does not exist" containerID="895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a"
Jul 28 18:39:51.858422 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.858175    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a"} err="failed to get container status \"895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a\": rpc error: code = NotFound desc = could not find container \"895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a\": container with ID starting with 895f6bad69b30599eccc93dcee40a46155f7933276a552ef363b8ff6ec2eed4a not found: ID does not exist"
Jul 28 18:39:51.858422 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.858203    2577 scope.go:117] "RemoveContainer" containerID="5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9"
Jul 28 18:39:51.862261 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:51.861216    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9\": container with ID starting with 5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9 not found: ID does not exist" containerID="5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9"
Jul 28 18:39:51.862261 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.861265    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9"} err="failed to get container status \"5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9\": rpc error: code = NotFound desc = could not find container \"5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9\": container with ID starting with 5a554da830a17335f590f977532c344718e9c567b46c0f62c78089ad8ed4f6e9 not found: ID does not exist"
Jul 28 18:39:51.909723 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:51.909687    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" path="/var/lib/kubelet/pods/59d0eb78-309e-4224-b975-a56d274f2dc6/volumes"
Jul 28 18:39:51.911443 ip-10-0-139-190 kubenswrapper[2577]: E0728 18:39:51.911407    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-czcp9\" is forbidden: User \"system:node:ip-10-0-139-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-lw9cf\": no relationship found between node 'ip-10-0-139-190.ec2.internal' and this object" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" pod="openshift-must-gather-lw9cf/must-gather-czcp9"
Jul 28 18:39:53.617796 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:53.617764    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-k4blr_337e7f5a-653e-4f40-a755-f57bae2e48b1/monitoring-plugin/0.log"
Jul 28 18:39:53.649330 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:53.649297    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fwgpv_f7120024-8fa9-4cd3-8e78-81df46bd8680/node-exporter/0.log"
Jul 28 18:39:53.673626 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:53.673594    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fwgpv_f7120024-8fa9-4cd3-8e78-81df46bd8680/kube-rbac-proxy/0.log"
Jul 28 18:39:53.697662 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:53.697631    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fwgpv_f7120024-8fa9-4cd3-8e78-81df46bd8680/init-textfile/0.log"
Jul 28 18:39:53.897195 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:53.897157    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rt26j_4d125b48-7cb2-4a74-9a60-e6549aaea6d4/kube-rbac-proxy-main/0.log"
Jul 28 18:39:53.922728 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:53.922688    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rt26j_4d125b48-7cb2-4a74-9a60-e6549aaea6d4/kube-rbac-proxy-self/0.log"
Jul 28 18:39:53.946366 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:53.946338    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-rt26j_4d125b48-7cb2-4a74-9a60-e6549aaea6d4/openshift-state-metrics/0.log"
Jul 28 18:39:53.993020 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:53.992983    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b221fe5e-9ba3-4b22-b088-2274244ab31f/prometheus/0.log"
Jul 28 18:39:54.020957 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.020927    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b221fe5e-9ba3-4b22-b088-2274244ab31f/config-reloader/0.log"
Jul 28 18:39:54.049760 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.049725    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b221fe5e-9ba3-4b22-b088-2274244ab31f/thanos-sidecar/0.log"
Jul 28 18:39:54.078686 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.078649    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b221fe5e-9ba3-4b22-b088-2274244ab31f/kube-rbac-proxy-web/0.log"
Jul 28 18:39:54.106151 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.106086    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b221fe5e-9ba3-4b22-b088-2274244ab31f/kube-rbac-proxy/0.log"
Jul 28 18:39:54.131023 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.130993    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b221fe5e-9ba3-4b22-b088-2274244ab31f/kube-rbac-proxy-thanos/0.log"
Jul 28 18:39:54.158365 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.158290    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b221fe5e-9ba3-4b22-b088-2274244ab31f/init-config-reloader/0.log"
Jul 28 18:39:54.198029 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.197993    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-545bt_13a8e82c-5c0c-4dd7-b155-08ac4c3927eb/prometheus-operator/0.log"
Jul 28 18:39:54.228728 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.228698    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-545bt_13a8e82c-5c0c-4dd7-b155-08ac4c3927eb/kube-rbac-proxy/0.log"
Jul 28 18:39:54.256813 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.256783    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-t9dcf_b3932887-752b-4af2-988d-dff0abd73767/prometheus-operator-admission-webhook/0.log"
Jul 28 18:39:54.393090 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.393060    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d4cbb8df6-2ljld_644a33d5-946e-4d11-a208-323e444fe48b/thanos-query/0.log"
Jul 28 18:39:54.420956 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.420935    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d4cbb8df6-2ljld_644a33d5-946e-4d11-a208-323e444fe48b/kube-rbac-proxy-web/0.log"
Jul 28 18:39:54.445712 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.445689    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d4cbb8df6-2ljld_644a33d5-946e-4d11-a208-323e444fe48b/kube-rbac-proxy/0.log"
Jul 28 18:39:54.468877 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.468850    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d4cbb8df6-2ljld_644a33d5-946e-4d11-a208-323e444fe48b/prom-label-proxy/0.log"
Jul 28 18:39:54.493052 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.493019    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d4cbb8df6-2ljld_644a33d5-946e-4d11-a208-323e444fe48b/kube-rbac-proxy-rules/0.log"
Jul 28 18:39:54.516495 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:54.516463    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5d4cbb8df6-2ljld_644a33d5-946e-4d11-a208-323e444fe48b/kube-rbac-proxy-metrics/0.log"
Jul 28 18:39:56.279107 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:56.279084    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/1.log"
Jul 28 18:39:56.285765 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:56.285741    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-vjx7p_a54fd9e3-e867-475a-ad77-09acb484c61e/console-operator/2.log"
Jul 28 18:39:57.325400 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.325371    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-7ktzj_a1a7a1d5-1f22-42b4-8776-a7a71eee2200/volume-data-source-validator/0.log"
Jul 28 18:39:57.471697 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.471653    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"]
Jul 28 18:39:57.472377 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.472352    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" containerName="copy"
Jul 28 18:39:57.472377 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.472378    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" containerName="copy"
Jul 28 18:39:57.472566 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.472438    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" containerName="gather"
Jul 28 18:39:57.472566 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.472447    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" containerName="gather"
Jul 28 18:39:57.472689 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.472597    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" containerName="gather"
Jul 28 18:39:57.472689 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.472609    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="59d0eb78-309e-4224-b975-a56d274f2dc6" containerName="copy"
Jul 28 18:39:57.480036 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.479989    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.481584 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.481557    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"]
Jul 28 18:39:57.563669 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.563631    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6twc\" (UniqueName: \"kubernetes.io/projected/208c15f1-5469-4ad6-810f-399df0731fd6-kube-api-access-x6twc\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.563978 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.563957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-proc\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.564161 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.564144    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-sys\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.564415 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.564390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-lib-modules\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.564518 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.564425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-podres\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.665206 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.665167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-proc\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.665441 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.665219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-sys\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.665441 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.665352    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-lib-modules\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.665441 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.665378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-podres\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.665441 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.665421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6twc\" (UniqueName: \"kubernetes.io/projected/208c15f1-5469-4ad6-810f-399df0731fd6-kube-api-access-x6twc\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.665891 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.665864    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-sys\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.666075 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.666054    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-lib-modules\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.666268 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.666059    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-proc\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.666398 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.666155    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/208c15f1-5469-4ad6-810f-399df0731fd6-podres\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.673558 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.673533    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6twc\" (UniqueName: \"kubernetes.io/projected/208c15f1-5469-4ad6-810f-399df0731fd6-kube-api-access-x6twc\") pod \"perf-node-gather-daemonset-h9jz4\" (UID: \"208c15f1-5469-4ad6-810f-399df0731fd6\") " pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.794177 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.794107    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:57.959005 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:57.958975    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"]
Jul 28 18:39:57.964142 ip-10-0-139-190 kubenswrapper[2577]: W0728 18:39:57.962181    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod208c15f1_5469_4ad6_810f_399df0731fd6.slice/crio-5c90fb1c4e564f54595617059f9a8eeb157c600623deccd1935da587fc70c036 WatchSource:0}: Error finding container 5c90fb1c4e564f54595617059f9a8eeb157c600623deccd1935da587fc70c036: Status 404 returned error can't find the container with id 5c90fb1c4e564f54595617059f9a8eeb157c600623deccd1935da587fc70c036
Jul 28 18:39:58.310195 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:58.310162    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7rfz_c648388c-85fe-4368-b838-b3fb9f356628/dns/0.log"
Jul 28 18:39:58.334232 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:58.334209    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d7rfz_c648388c-85fe-4368-b838-b3fb9f356628/kube-rbac-proxy/0.log"
Jul 28 18:39:58.455975 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:58.455946    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mz5qt_c40a55bc-b4ed-4df5-8c3c-a939e7989b12/dns-node-resolver/0.log"
Jul 28 18:39:58.866436 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:58.866399    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4" event={"ID":"208c15f1-5469-4ad6-810f-399df0731fd6","Type":"ContainerStarted","Data":"6e2850c9dad3db2f59202546718b710719f41c1d1ac066d57df039995cd10061"}
Jul 28 18:39:58.866638 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:58.866465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4" event={"ID":"208c15f1-5469-4ad6-810f-399df0731fd6","Type":"ContainerStarted","Data":"5c90fb1c4e564f54595617059f9a8eeb157c600623deccd1935da587fc70c036"}
Jul 28 18:39:58.866638 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:58.866495    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:39:58.883811 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:58.883767    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4" podStartSLOduration=1.883749645 podStartE2EDuration="1.883749645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:39:58.882959365 +0000 UTC m=+5357.526928867" watchObservedRunningTime="2026-07-28 18:39:58.883749645 +0000 UTC m=+5357.527719186"
Jul 28 18:39:59.004984 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:59.004957    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jht5g_c023b43e-30f9-4d56-aa4f-50f5e86724cf/node-ca/0.log"
Jul 28 18:39:59.701936 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:39:59.701909    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64cc6c558f-tgnb7_eb22b701-7530-4ee7-9116-29d6dfde5b0a/router/0.log"
Jul 28 18:40:00.085908 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:00.085832    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wmsx9_01cc0f78-cb16-4263-8ed2-0cb78bb0ef3c/serve-healthcheck-canary/0.log"
Jul 28 18:40:00.580093 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:00.580067    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ql87_6ef96cb6-2f40-4f9a-979a-c32a05499500/kube-rbac-proxy/0.log"
Jul 28 18:40:00.602638 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:00.602617    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ql87_6ef96cb6-2f40-4f9a-979a-c32a05499500/exporter/0.log"
Jul 28 18:40:00.625364 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:00.625342    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2ql87_6ef96cb6-2f40-4f9a-979a-c32a05499500/extractor/0.log"
Jul 28 18:40:03.319648 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:03.319617    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-snxwh_2584fe5c-f355-4eb8-93b5-64ff9ed74149/s3-init/0.log"
Jul 28 18:40:03.337553 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:03.337535    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-snxwh_2584fe5c-f355-4eb8-93b5-64ff9ed74149/download-hf-model/0.log"
Jul 28 18:40:03.367548 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:03.367522    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7lhbf_575d67d0-f00c-4b74-a316-99a8d0c8f4e5/s3-tls-init-custom/0.log"
Jul 28 18:40:03.385980 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:03.385956    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-7lhbf_575d67d0-f00c-4b74-a316-99a8d0c8f4e5/download-hf-model/0.log"
Jul 28 18:40:03.415156 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:03.415138    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-sfwbj_710814b1-2021-4845-96b8-6352b64b667c/s3-tls-init-serving/0.log"
Jul 28 18:40:03.432281 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:03.432260    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-sfwbj_710814b1-2021-4845-96b8-6352b64b667c/download-hf-model/0.log"
Jul 28 18:40:04.886200 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:04.886153    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kgbts/perf-node-gather-daemonset-h9jz4"
Jul 28 18:40:07.720217 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:07.720176    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9vrjx_1f91a8ac-7750-4476-a1a9-3a3f49844fab/migrator/0.log"
Jul 28 18:40:07.741978 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:07.741955    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-9vrjx_1f91a8ac-7750-4476-a1a9-3a3f49844fab/graceful-termination/0.log"
Jul 28 18:40:09.062186 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.062161    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9wfsw_a5d86e2e-21d5-4499-9468-1f98e0d63a2c/kube-multus/0.log"
Jul 28 18:40:09.248676 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.248649    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j7k5l_e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:40:09.272650 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.272568    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j7k5l_e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31/egress-router-binary-copy/0.log"
Jul 28 18:40:09.295089 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.295063    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j7k5l_e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31/cni-plugins/0.log"
Jul 28 18:40:09.352462 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.352428    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j7k5l_e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31/bond-cni-plugin/0.log"
Jul 28 18:40:09.372518 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.372497    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j7k5l_e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31/routeoverride-cni/0.log"
Jul 28 18:40:09.397309 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.397285    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j7k5l_e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31/whereabouts-cni-bincopy/0.log"
Jul 28 18:40:09.419803 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.419773    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-j7k5l_e0dff0f4-1fbd-46eb-9617-c7e45ee2ab31/whereabouts-cni/0.log"
Jul 28 18:40:09.630548 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.630461    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6czkd_9cdd97a5-b1fd-4756-9933-1a551767c9c6/network-metrics-daemon/0.log"
Jul 28 18:40:09.652160 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:09.652105    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6czkd_9cdd97a5-b1fd-4756-9933-1a551767c9c6/kube-rbac-proxy/0.log"
Jul 28 18:40:10.467765 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:10.467736    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9nr_c7cc17ed-e2a2-4226-9287-63c1c56238fc/ovn-controller/0.log"
Jul 28 18:40:10.486926 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:10.486898    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9nr_c7cc17ed-e2a2-4226-9287-63c1c56238fc/ovn-acl-logging/0.log"
Jul 28 18:40:10.511413 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:10.511391    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9nr_c7cc17ed-e2a2-4226-9287-63c1c56238fc/kube-rbac-proxy-node/0.log"
Jul 28 18:40:10.532704 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:10.532675    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9nr_c7cc17ed-e2a2-4226-9287-63c1c56238fc/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:40:10.555731 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:10.555713    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9nr_c7cc17ed-e2a2-4226-9287-63c1c56238fc/northd/0.log"
Jul 28 18:40:10.576842 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:10.576826    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9nr_c7cc17ed-e2a2-4226-9287-63c1c56238fc/nbdb/0.log"
Jul 28 18:40:10.598714 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:10.598697    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9nr_c7cc17ed-e2a2-4226-9287-63c1c56238fc/sbdb/0.log"
Jul 28 18:40:10.658016 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:10.657990    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ck9nr_c7cc17ed-e2a2-4226-9287-63c1c56238fc/ovnkube-controller/0.log"
Jul 28 18:40:12.082943 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:12.082905    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-b5btq_fd1d9d5f-6d34-4b5c-909a-56b7532fc4b3/network-check-target-container/0.log"
Jul 28 18:40:12.988329 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:12.988297    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5m4kt_8b7cd99b-bfab-4d37-a1b4-9d39302ba332/iptables-alerter/0.log"
Jul 28 18:40:13.622322 ip-10-0-139-190 kubenswrapper[2577]: I0728 18:40:13.622291    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-fk7jp_facca3dd-945a-42d5-a32a-b8e4de3f7108/tuned/0.log"