May 25 11:17:31.230848 ip-10-0-128-161 systemd[1]: Starting Kubernetes Kubelet...
May 25 11:17:31.715702 ip-10-0-128-161 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:31.715702 ip-10-0-128-161 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
May 25 11:17:31.715702 ip-10-0-128-161 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:31.715702 ip-10-0-128-161 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
May 25 11:17:31.715702 ip-10-0-128-161 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
May 25 11:17:31.719128 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.719038    2556 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
May 25 11:17:31.721931 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721916    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:31.721931 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721931    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721934    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721938    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721941    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721944    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721947    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721950    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721952    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721955    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721958    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721961    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721964    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721967    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721970    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721973    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721975    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721980    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721984    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721987    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:31.721993 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721990    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721993    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721996    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.721999    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722002    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722005    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722007    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722010    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722013    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722015    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722018    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722020    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722023    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722025    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722028    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722030    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722033    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722036    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722038    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722041    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:31.722430 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722044    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722048    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722050    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722053    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722056    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722059    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722061    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722064    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722067    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722069    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722072    2556 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722074    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722077    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722079    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722082    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722085    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722088    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722091    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722093    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722096    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:31.722945 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722098    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722101    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722103    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722106    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722108    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722111    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722113    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722116    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722119    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722122    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722124    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722127    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722130    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722132    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722135    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722137    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722140    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722143    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722146    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722150    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:31.723437 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722152    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722154    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722157    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722159    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722162    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722165    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722568    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722574    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722577    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722580    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722582    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722585    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722588    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722590    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722593    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722596    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722599    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722601    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722604    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722607    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:31.724008 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722609    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722612    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722615    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722617    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722620    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722622    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722625    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722627    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722630    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722633    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722635    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722638    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722641    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722643    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722646    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722648    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722651    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722653    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722657    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722660    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:31.724535 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722662    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722665    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722667    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722670    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722672    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722675    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722677    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722680    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722683    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722685    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722688    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722691    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722693    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722696    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722698    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722701    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722703    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722706    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722709    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722711    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:31.725020 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722714    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722717    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722719    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722722    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722724    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722727    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722729    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722732    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722734    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722736    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722741    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722744    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722746    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722749    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722752    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722754    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722763    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722766    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722769    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:31.725520 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722772    2556 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722775    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722777    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722780    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722782    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722785    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722787    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722791    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722794    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722797    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722800    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722802    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.722805    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724274    2556 flags.go:64] FLAG: --address="0.0.0.0"
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724283    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724295    2556 flags.go:64] FLAG: --anonymous-auth="true"
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724301    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724306    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724309    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724313    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724318    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
May 25 11:17:31.725985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724321    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724324    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724328    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724332    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724335    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724338    2556 flags.go:64] FLAG: --cgroup-root=""
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724341    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724344    2556 flags.go:64] FLAG: --client-ca-file=""
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724346    2556 flags.go:64] FLAG: --cloud-config=""
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724349    2556 flags.go:64] FLAG: --cloud-provider="external"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724352    2556 flags.go:64] FLAG: --cluster-dns="[]"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724356    2556 flags.go:64] FLAG: --cluster-domain=""
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724359    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724362    2556 flags.go:64] FLAG: --config-dir=""
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724365    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724368    2556 flags.go:64] FLAG: --container-log-max-files="5"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724372    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724375    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724378    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724381    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724384    2556 flags.go:64] FLAG: --contention-profiling="false"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724387    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724390    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724393    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724396    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
May 25 11:17:31.726498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724400    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724403    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724406    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724410    2556 flags.go:64] FLAG: --enable-load-reader="false"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724413    2556 flags.go:64] FLAG: --enable-server="true"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724416    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724421    2556 flags.go:64] FLAG: --event-burst="100"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724424    2556 flags.go:64] FLAG: --event-qps="50"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724427    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724430    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724433    2556 flags.go:64] FLAG: --eviction-hard=""
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724437    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724440    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724443    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724446    2556 flags.go:64] FLAG: --eviction-soft=""
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724448    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724451    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724454    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724457    2556 flags.go:64] FLAG: --experimental-mounter-path=""
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724460    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724463    2556 flags.go:64] FLAG: --fail-swap-on="true"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724466    2556 flags.go:64] FLAG: --feature-gates=""
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724470    2556 flags.go:64] FLAG: --file-check-frequency="20s"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724473    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724476    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
May 25 11:17:31.727120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724493    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724496    2556 flags.go:64] FLAG: --healthz-port="10248"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724500    2556 flags.go:64] FLAG: --help="false"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724503    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-128-161.ec2.internal"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724506    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724510    2556 flags.go:64] FLAG: --http-check-frequency="20s"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724513    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724516    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724520    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724523    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724525    2556 flags.go:64] FLAG: --image-service-endpoint=""
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724529    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724532    2556 flags.go:64] FLAG: --kube-api-burst="100"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724535    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724539    2556 flags.go:64] FLAG: --kube-api-qps="50"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724542    2556 flags.go:64] FLAG: --kube-reserved=""
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724545    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724548    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724551    2556 flags.go:64] FLAG: --kubelet-cgroups=""
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724554    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724557    2556 flags.go:64] FLAG: --lock-file=""
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724560    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724563    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724566    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
May 25 11:17:31.727734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724571    2556 flags.go:64] FLAG: --log-json-split-stream="false"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724574    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724577    2556 flags.go:64] FLAG: --log-text-split-stream="false"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724579    2556 flags.go:64] FLAG: --logging-format="text"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724582    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724586    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724588    2556 flags.go:64] FLAG: --manifest-url=""
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724591    2556 flags.go:64] FLAG: --manifest-url-header=""
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724596    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724599    2556 flags.go:64] FLAG: --max-open-files="1000000"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724603    2556 flags.go:64] FLAG: --max-pods="110"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724606    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724609    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724612    2556 flags.go:64] FLAG: --memory-manager-policy="None"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724615    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724619    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724622    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724625    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724632    2556 flags.go:64] FLAG: --node-status-max-images="50"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724636    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724639    2556 flags.go:64] FLAG: --oom-score-adj="-999"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724642    2556 flags.go:64] FLAG: --pod-cidr=""
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724645    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc"
May 25 11:17:31.728298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724650    2556 flags.go:64] FLAG: --pod-manifest-path=""
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724653    2556 flags.go:64] FLAG: --pod-max-pids="-1"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724659    2556 flags.go:64] FLAG: --pods-per-core="0"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724662    2556 flags.go:64] FLAG: --port="10250"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724665    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724668    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08c4b1b5c7992d1e4"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724671    2556 flags.go:64] FLAG: --qos-reserved=""
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724674    2556 flags.go:64] FLAG: --read-only-port="10255"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724677    2556 flags.go:64] FLAG: --register-node="true"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724680    2556 flags.go:64] FLAG: --register-schedulable="true"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724683    2556 flags.go:64] FLAG: --register-with-taints=""
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724687    2556 flags.go:64] FLAG: --registry-burst="10"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724690    2556 flags.go:64] FLAG: --registry-qps="5"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724692    2556 flags.go:64] FLAG: --reserved-cpus=""
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724695    2556 flags.go:64] FLAG: --reserved-memory=""
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724699    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724702    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724704    2556 flags.go:64] FLAG: --rotate-certificates="false"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724707    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724710    2556 flags.go:64] FLAG: --runonce="false"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724713    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724716    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724719    2556 flags.go:64] FLAG: --seccomp-default="false"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724722    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724725    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724728    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
May 25 11:17:31.728906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724731    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724734    2556 flags.go:64] FLAG: --storage-driver-password="root"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724737    2556 flags.go:64] FLAG: --storage-driver-secure="false"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724740    2556 flags.go:64] FLAG: --storage-driver-table="stats"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724747    2556 flags.go:64] FLAG: --storage-driver-user="root"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724750    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724753    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724756    2556 flags.go:64] FLAG: --system-cgroups=""
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724760    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724765    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724768    2556 flags.go:64] FLAG: --tls-cert-file=""
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724771    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724775    2556 flags.go:64] FLAG: --tls-min-version=""
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724777    2556 flags.go:64] FLAG: --tls-private-key-file=""
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724781    2556 flags.go:64] FLAG: --topology-manager-policy="none"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724784    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724787    2556 flags.go:64] FLAG: --topology-manager-scope="container"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724790    2556 flags.go:64] FLAG: --v="2"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724794    2556 flags.go:64] FLAG: --version="false"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724798    2556 flags.go:64] FLAG: --vmodule=""
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724802    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.724805    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724894    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724897    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:31.729537 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724900    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724904    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724907    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724910    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724913    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724917    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724920    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724923    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724926    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724929    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724932    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724935    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724937    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724940    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724943    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724946    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724952    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724955    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724958    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724961    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:31.730099 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724963    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724966    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724968    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724971    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724973    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724976    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724978    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724981    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724983    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724986    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724988    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724991    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724994    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724996    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.724998    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725002    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725006    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725008    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725011    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:31.730647 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725014    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725016    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725019    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725022    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725025    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725027    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725030    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725033    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725036    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725040    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725043    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725045    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725048    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725050    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725052    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725055    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725057    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725060    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725062    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725065    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:31.731159 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725067    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725070    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725072    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725075    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725077    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725080    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725082    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725085    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725087    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725090    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725092    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725095    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725098    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725100    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725103    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725106    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725108    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725111    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725114    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:31.731970 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725116    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:31.732436 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725119    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:31.732436 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725123    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:31.732436 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725125    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:31.732436 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725128    2556 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:31.732436 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.725130    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:31.732436 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.725136    2556 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:31.733562 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.733544    2556 server.go:530] "Kubelet version" kubeletVersion="v1.33.11"
May 25 11:17:31.733562 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.733563    2556 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733610    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733615    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733618    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733622    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733625    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733627    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733630    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733634    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733636    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733639    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733642    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733645    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733647    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733650    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733653    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733655    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733658    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733660    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733663    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:31.733689 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733666    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733668    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733671    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733674    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733677    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733680    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733683    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733685    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733688    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733691    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733693    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733696    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733699    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733702    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733705    2556 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733709    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733713    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733716    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733718    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:31.734171 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733721    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733724    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733726    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733729    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733732    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733734    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733736    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733739    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733741    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733744    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733746    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733749    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733752    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733754    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733757    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733761    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733764    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733766    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733769    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733772    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:31.734655 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733774    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733777    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733779    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733782    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733784    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733787    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733789    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733792    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733794    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733797    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733799    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733802    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733804    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733807    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733809    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733813    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733818    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733821    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733824    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:31.735150 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733827    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733829    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733832    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733834    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733837    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733839    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733842    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733844    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733847    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.733853    2556 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733943    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733948    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733951    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733954    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733958    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
May 25 11:17:31.735638 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733961    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733964    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733966    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733969    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733972    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733975    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733978    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733981    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733983    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733986    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733989    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733991    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733994    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733997    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.733999    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734002    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734004    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734007    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734010    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734012    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734015    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
May 25 11:17:31.736007 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734017    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734020    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734023    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734025    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734028    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734031    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734034    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734036    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734039    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734041    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734044    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734046    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734049    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734051    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734054    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734056    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734059    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734061    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734064    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734066    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
May 25 11:17:31.736551 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734069    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734072    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734074    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734077    2556 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734079    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734082    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734084    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734087    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734090    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734093    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734095    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734098    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734100    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734103    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734105    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734108    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734111    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734114    2556 feature_gate.go:328] unrecognized feature gate: Example2
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734116    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734119    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
May 25 11:17:31.737034 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734121    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734124    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734127    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734129    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734132    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734135    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734138    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734140    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734143    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734145    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734148    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734150    2556 feature_gate.go:328] unrecognized feature gate: Example
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734153    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734155    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734158    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734162    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734166    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734170    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734173    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
May 25 11:17:31.737536 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:31.734176    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
May 25 11:17:31.738010 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.734181    2556 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]}
May 25 11:17:31.738010 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.735047    2556 server.go:962] "Client rotation is on, will bootstrap in background"
May 25 11:17:31.738010 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.737905    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
May 25 11:17:31.738933 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.738922    2556 server.go:1019] "Starting client certificate rotation"
May 25 11:17:31.739047 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.739029    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:31.739104 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.739069    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
May 25 11:17:31.767212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.767192    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:31.770669 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.770651    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
May 25 11:17:31.786459 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.786441    2556 log.go:25] "Validated CRI v1 runtime API"
May 25 11:17:31.792337 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.792320    2556 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:31.792522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.792509    2556 log.go:25] "Validated CRI v1 image API"
May 25 11:17:31.793778 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.793757    2556 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
May 25 11:17:31.799560 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.799539    2556 fs.go:135] Filesystem UUIDs: map[56fa1a5e-358d-4603-876a-f110f36cfcdb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b96cd8b2-d715-4fab-a91d-a3f6e2422572:/dev/nvme0n1p4]
May 25 11:17:31.799632 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.799558    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
May 25 11:17:31.803646 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.803545    2556 manager.go:217] Machine: {Timestamp:2026-05-25 11:17:31.803052687 +0000 UTC m=+0.433213488 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098312 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a6f4368bbf31e7256e61071986ab2 SystemUUID:ec2a6f43-68bb-f31e-7256-e61071986ab2 BootID:f8535439-c853-40a8-86e2-a645900de0f8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:82:dd:af:73:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:82:dd:af:73:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ea:8c:82:d0:fb:8a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
May 25 11:17:31.804526 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.804516    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
May 25 11:17:31.804600 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.804588    2556 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
May 25 11:17:31.806724 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.806698    2556 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
May 25 11:17:31.806849 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.806727    2556 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-161.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
May 25 11:17:31.806897 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.806858    2556 topology_manager.go:138] "Creating topology manager with none policy"
May 25 11:17:31.806897 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.806866    2556 container_manager_linux.go:306] "Creating device plugin manager"
May 25 11:17:31.806897 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.806875    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:31.807764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.807754    2556 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
May 25 11:17:31.809715 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.809706    2556 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:31.809954 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.809944    2556 server.go:1267] "Using root directory" path="/var/lib/kubelet"
May 25 11:17:31.812589 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.812579    2556 kubelet.go:491] "Attempting to sync node with API server"
May 25 11:17:31.812623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.812592    2556 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests"
May 25 11:17:31.812623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.812603    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
May 25 11:17:31.812623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.812611    2556 kubelet.go:397] "Adding apiserver pod source"
May 25 11:17:31.812623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.812620    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
May 25 11:17:31.813818 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.813806    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:31.813864 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.813825    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
May 25 11:17:31.814412 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.814399    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gdczv"
May 25 11:17:31.816694 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.816673    2556 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1"
May 25 11:17:31.818436 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.818420    2556 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
May 25 11:17:31.819563 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819549    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
May 25 11:17:31.819649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819570    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
May 25 11:17:31.819649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819578    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
May 25 11:17:31.819649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819586    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
May 25 11:17:31.819649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819597    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
May 25 11:17:31.819649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819605    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret"
May 25 11:17:31.819649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819613    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
May 25 11:17:31.819649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819623    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
May 25 11:17:31.819649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819633    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc"
May 25 11:17:31.819649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819641    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
May 25 11:17:31.819922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.819663    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected"
May 25 11:17:31.820082 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.820071    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
May 25 11:17:31.821282 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.821272    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi"
May 25 11:17:31.821347 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.821284    2556 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image"
May 25 11:17:31.822819 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.822799    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gdczv"
May 25 11:17:31.824329 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.824312    2556 watchdog_linux.go:99] "Systemd watchdog is not enabled"
May 25 11:17:31.824413 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.824347    2556 server.go:1295] "Started kubelet"
May 25 11:17:31.824512 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.824467    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
May 25 11:17:31.824623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.824513    2556 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
May 25 11:17:31.824623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.824570    2556 server_v1.go:47] "podresources" method="list" useActivePods=true
May 25 11:17:31.824975 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:31.824953    2556 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-161.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
May 25 11:17:31.825060 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.825006    2556 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-161.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
May 25 11:17:31.825122 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:31.825101    2556 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
May 25 11:17:31.825170 ip-10-0-128-161 systemd[1]: Started Kubernetes Kubelet.
May 25 11:17:31.826975 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.826962    2556 server.go:317] "Adding debug handlers to kubelet server"
May 25 11:17:31.828881 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.828860    2556 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
May 25 11:17:31.835217 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.835198    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
May 25 11:17:31.835356 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:31.835332    2556 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
May 25 11:17:31.836580 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.836565    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
May 25 11:17:31.836728 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.836710    2556 volume_manager.go:295] "The desired_state_of_world populator starts"
May 25 11:17:31.836728 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.836727    2556 volume_manager.go:297] "Starting Kubelet Volume Manager"
May 25 11:17:31.836868 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.836811    2556 desired_state_of_world_populator.go:150] "Desired state populator starts to run"
May 25 11:17:31.836922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.836878    2556 reconstruct.go:97] "Volume reconstruction finished"
May 25 11:17:31.836922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.836887    2556 reconciler.go:26] "Reconciler: start to sync state"
May 25 11:17:31.837155 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.837141    2556 factory.go:55] Registering systemd factory
May 25 11:17:31.837205 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.837164    2556 factory.go:223] Registration of the systemd container factory successfully
May 25 11:17:31.837386 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.837373    2556 factory.go:153] Registering CRI-O factory
May 25 11:17:31.837433 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.837388    2556 factory.go:223] Registration of the crio container factory successfully
May 25 11:17:31.837433 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.837431    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
May 25 11:17:31.837538 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.837450    2556 factory.go:103] Registering Raw factory
May 25 11:17:31.837538 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.837462    2556 manager.go:1196] Started watching for new ooms in manager
May 25 11:17:31.837538 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:31.837529    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:31.837932 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.837878    2556 manager.go:319] Starting recovery of all containers
May 25 11:17:31.840019 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.840001    2556 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:31.842722 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:31.842575    2556 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-161.ec2.internal\" not found" node="ip-10-0-128-161.ec2.internal"
May 25 11:17:31.847269 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.847256    2556 manager.go:324] Recovery completed
May 25 11:17:31.851574 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.851561    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:31.853447 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.853432    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:31.853528 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.853459    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:31.853528 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.853469    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:31.853968 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.853944    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
May 25 11:17:31.854030 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.853969    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
May 25 11:17:31.854030 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.853987    2556 state_mem.go:36] "Initialized new in-memory state store"
May 25 11:17:31.856891 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.856877    2556 policy_none.go:49] "None policy: Start"
May 25 11:17:31.856972 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.856896    2556 memory_manager.go:186] "Starting memorymanager" policy="None"
May 25 11:17:31.856972 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.856910    2556 state_mem.go:35] "Initializing new in-memory state store"
May 25 11:17:31.898077 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.898062    2556 manager.go:341] "Starting Device Plugin manager"
May 25 11:17:31.898167 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:31.898095    2556 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
May 25 11:17:31.898167 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.898107    2556 server.go:85] "Starting device plugin registration server"
May 25 11:17:31.898328 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.898316    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
May 25 11:17:31.898381 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.898329    2556 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
May 25 11:17:31.898426 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.898397    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
May 25 11:17:31.898520 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.898478    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
May 25 11:17:31.898520 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.898512    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
May 25 11:17:31.899106 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:31.899068    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
May 25 11:17:31.899192 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:31.899113    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:31.974469 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.974416    2556 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4"
May 25 11:17:31.975613 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.975598    2556 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6"
May 25 11:17:31.975694 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.975622    2556 status_manager.go:230] "Starting to sync pod status with apiserver"
May 25 11:17:31.975694 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.975638    2556 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
May 25 11:17:31.975694 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.975644    2556 kubelet.go:2451] "Starting kubelet main sync loop"
May 25 11:17:31.975694 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:31.975678    2556 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
May 25 11:17:31.979985 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.979966    2556 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:31.998574 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.998562    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:31.999383 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.999368    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:31.999463 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.999392    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:31.999463 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.999403    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:31.999463 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:31.999423    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-161.ec2.internal"
May 25 11:17:32.007271 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.007255    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-161.ec2.internal"
May 25 11:17:32.007315 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.007273    2556 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-161.ec2.internal\": node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.076132 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.076109    2556 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"]
May 25 11:17:32.076204 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.076165    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:32.076883 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.076867    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:32.076952 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.076893    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:32.076952 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.076902    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:32.078127 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.078116    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:32.078282 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.078270    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.078316 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.078296    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:32.078786 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.078769    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:32.078867 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.078772    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:32.078867 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.078826    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:32.078867 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.078837    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:32.078867 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.078801    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:32.078867 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.078867    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:32.079948 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.079935    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.079995 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.079957    2556 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach"
May 25 11:17:32.080556 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.080541    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientMemory"
May 25 11:17:32.080620 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.080566    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasNoDiskPressure"
May 25 11:17:32.080620 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.080582    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeHasSufficientPID"
May 25 11:17:32.085172 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.085155    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.096067 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.096047    2556 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-161.ec2.internal\" not found" node="ip-10-0-128-161.ec2.internal"
May 25 11:17:32.099710 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.099695    2556 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-161.ec2.internal\" not found" node="ip-10-0-128-161.ec2.internal"
May 25 11:17:32.139476 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.139455    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b79a830b6f70433eb778f4f12409255-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"8b79a830b6f70433eb778f4f12409255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.139590 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.139501    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b79a830b6f70433eb778f4f12409255-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"8b79a830b6f70433eb778f4f12409255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.139590 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.139530    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3204d4e276a6036a6def62b77dcf7af-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.ec2.internal\" (UID: \"b3204d4e276a6036a6def62b77dcf7af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.186016 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.185995    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.240686 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.240642    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3204d4e276a6036a6def62b77dcf7af-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.ec2.internal\" (UID: \"b3204d4e276a6036a6def62b77dcf7af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.240686 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.240602    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b3204d4e276a6036a6def62b77dcf7af-config\") pod \"kube-apiserver-proxy-ip-10-0-128-161.ec2.internal\" (UID: \"b3204d4e276a6036a6def62b77dcf7af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.240826 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.240703    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b79a830b6f70433eb778f4f12409255-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"8b79a830b6f70433eb778f4f12409255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.240826 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.240725    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b79a830b6f70433eb778f4f12409255-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"8b79a830b6f70433eb778f4f12409255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.240826 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.240762    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b79a830b6f70433eb778f4f12409255-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"8b79a830b6f70433eb778f4f12409255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.240826 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.240818    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8b79a830b6f70433eb778f4f12409255-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal\" (UID: \"8b79a830b6f70433eb778f4f12409255\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.286079 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.286057    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.386554 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.386531    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.399671 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.399654    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.402244 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.402231    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
May 25 11:17:32.487031 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.487007    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.587454 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.587404    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.687862 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.687826    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.739137 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.739113    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
May 25 11:17:32.739728 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.739257    2556 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:32.739728 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.739275    2556 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
May 25 11:17:32.788565 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.788541    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.824997 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.824965    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-24 11:12:31 +0000 UTC" deadline="2027-12-15 08:49:00.217737051 +0000 UTC"
May 25 11:17:32.824997 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.824995    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13653h31m27.39274461s"
May 25 11:17:32.835541 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.835520    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
May 25 11:17:32.844724 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.844677    2556 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
May 25 11:17:32.872238 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.872222    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9652n"
May 25 11:17:32.879477 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.879456    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9652n"
May 25 11:17:32.881222 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:32.881197    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3204d4e276a6036a6def62b77dcf7af.slice/crio-aca2c435b394c17209cccb5e4c701d77cf6460aac502c28c975b42cb7eecb17c WatchSource:0}: Error finding container aca2c435b394c17209cccb5e4c701d77cf6460aac502c28c975b42cb7eecb17c: Status 404 returned error can't find the container with id aca2c435b394c17209cccb5e4c701d77cf6460aac502c28c975b42cb7eecb17c
May 25 11:17:32.881560 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:32.881534    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b79a830b6f70433eb778f4f12409255.slice/crio-f8ad73b5f60ec091c284f858a98dd3fa72fa693b4aa8c5bf0b49357a36936981 WatchSource:0}: Error finding container f8ad73b5f60ec091c284f858a98dd3fa72fa693b4aa8c5bf0b49357a36936981: Status 404 returned error can't find the container with id f8ad73b5f60ec091c284f858a98dd3fa72fa693b4aa8c5bf0b49357a36936981
May 25 11:17:32.884698 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.884682    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:17:32.889342 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.889324    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:32.978668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.978629    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" event={"ID":"8b79a830b6f70433eb778f4f12409255","Type":"ContainerStarted","Data":"f8ad73b5f60ec091c284f858a98dd3fa72fa693b4aa8c5bf0b49357a36936981"}
May 25 11:17:32.979474 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:32.979455    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal" event={"ID":"b3204d4e276a6036a6def62b77dcf7af","Type":"ContainerStarted","Data":"aca2c435b394c17209cccb5e4c701d77cf6460aac502c28c975b42cb7eecb17c"}
May 25 11:17:32.989603 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:32.989587    2556 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-128-161.ec2.internal\" not found"
May 25 11:17:33.036118 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.036103    2556 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:33.037067 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.037053    2556 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal"
May 25 11:17:33.048778 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.048763    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:33.050624 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.050613    2556 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal"
May 25 11:17:33.063173 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.063150    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
May 25 11:17:33.340965 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.340890    2556 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:33.814562 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.814494    2556 apiserver.go:52] "Watching apiserver"
May 25 11:17:33.822853 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.822835    2556 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
May 25 11:17:33.823763 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.823734    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-8w9qg","kube-system/konnectivity-agent-n2d2h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs","openshift-image-registry/node-ca-v2shg","openshift-network-diagnostics/network-check-target-2j6dr","openshift-network-operator/iptables-alerter-879wd","openshift-ovn-kubernetes/ovnkube-node-794p7","kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9sv9g","openshift-dns/node-resolver-hp2v9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal","openshift-multus/multus-additional-cni-plugins-sh48r","openshift-multus/multus-hkkfj"]
May 25 11:17:33.825328 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.825298    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.826297 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.826272    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:33.828493 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.828457    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
May 25 11:17:33.828591 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.828575    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.828740 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.828713    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
May 25 11:17:33.829051 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.829032    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mzwtp\""
May 25 11:17:33.829129 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.829116    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
May 25 11:17:33.830364 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.830344    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
May 25 11:17:33.830456 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.830350    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
May 25 11:17:33.830581 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.830566    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kdhkw\""
May 25 11:17:33.830664 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.830650    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
May 25 11:17:33.830779 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.830766    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
May 25 11:17:33.831314 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.831297    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.831392 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.831377    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:33.831493 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:33.831447    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:33.831607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.831538    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
May 25 11:17:33.833091 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.831776    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
May 25 11:17:33.833091 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.831811    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
May 25 11:17:33.833091 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.831821    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nftj\""
May 25 11:17:33.833091 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.831928    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
May 25 11:17:33.833091 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.832587    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.833636 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.833618    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
May 25 11:17:33.833735 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.833719    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
May 25 11:17:33.833795 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.833741    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8j5dk\""
May 25 11:17:33.834420 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.834056    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
May 25 11:17:33.835730 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.835711    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:33.835808 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.835755    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
May 25 11:17:33.835808 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.835799    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-b8b5s\""
May 25 11:17:33.836318 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.836294    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:33.836408 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:33.836379    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:33.837967 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.837055    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
May 25 11:17:33.838055 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.837996    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.838496 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.838192    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.839846 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.839824    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:33.841018 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.840999    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.842940 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.842920    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
May 25 11:17:33.843123 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.843101    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
May 25 11:17:33.843204 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.843150    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
May 25 11:17:33.843287 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.843270    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
May 25 11:17:33.843379 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.843368    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9k7hp\""
May 25 11:17:33.843745 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.843729    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gwbsw\""
May 25 11:17:33.844508 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.844442    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
May 25 11:17:33.844691 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.844637    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
May 25 11:17:33.844780 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.844750    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
May 25 11:17:33.844841 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.844808    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
May 25 11:17:33.844977 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.844951    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
May 25 11:17:33.845041 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.844992    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lh5l8\""
May 25 11:17:33.845041 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.845021    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9d65h\""
May 25 11:17:33.845136 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.845064    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
May 25 11:17:33.850795 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850777    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-node-log\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.850889 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850803    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-var-lib-openvswitch\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.850889 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850827    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-run-ovn-kubernetes\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.850889 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-sys-fs\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.850889 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850868    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9607124d-82db-4c28-a3a4-5a49d4603955-ovnkube-config\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851056 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850891    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-kubernetes\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.851056 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850914    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec6f6e8-33f0-43d3-b580-0b98f54b3bf3-tmp-dir\") pod \"node-resolver-hp2v9\" (UID: \"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3\") " pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.851056 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850936    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9607124d-82db-4c28-a3a4-5a49d4603955-ovnkube-script-lib\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851056 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850951    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90-iptables-alerter-script\") pod \"iptables-alerter-879wd\" (UID: \"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90\") " pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.851056 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850965    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2l9\" (UniqueName: \"kubernetes.io/projected/37262c21-a42a-4ff7-90c7-bbe50a05d68b-kube-api-access-jd2l9\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:33.851056 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.850981    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-run-netns\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851056 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851023    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-etc-openvswitch\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-sysconfig\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851124    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9607124d-82db-4c28-a3a4-5a49d4603955-env-overrides\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851139    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-socket-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af6ff664-f55f-4650-b65a-3816ace0d8c0-tmp\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851181    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-cni-bin\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851200    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c447b\" (UniqueName: \"kubernetes.io/projected/9607124d-82db-4c28-a3a4-5a49d4603955-kube-api-access-c447b\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851216    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851260    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdef210b-a23e-438e-a7d9-7978931d243e-host\") pod \"node-ca-v2shg\" (UID: \"fdef210b-a23e-438e-a7d9-7978931d243e\") " pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851283    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-kubelet\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851323    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb296f1f-d101-4bd7-a981-3803bc01c51a-agent-certs\") pod \"konnectivity-agent-n2d2h\" (UID: \"fb296f1f-d101-4bd7-a981-3803bc01c51a\") " pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851350    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-device-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.851376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5vsx\" (UniqueName: \"kubernetes.io/projected/9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90-kube-api-access-b5vsx\") pod \"iptables-alerter-879wd\" (UID: \"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90\") " pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851392    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-sysctl-d\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851432    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851454    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmblr\" (UniqueName: \"kubernetes.io/projected/aec6f6e8-33f0-43d3-b580-0b98f54b3bf3-kube-api-access-vmblr\") pod \"node-resolver-hp2v9\" (UID: \"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3\") " pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851500    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-log-socket\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851524    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-host\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851545    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-run-ovn\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851580    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9607124d-82db-4c28-a3a4-5a49d4603955-ovn-node-metrics-cert\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851599    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fdef210b-a23e-438e-a7d9-7978931d243e-serviceca\") pod \"node-ca-v2shg\" (UID: \"fdef210b-a23e-438e-a7d9-7978931d243e\") " pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851618    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-slash\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851643    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-run-openvswitch\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851669    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90-host-slash\") pod \"iptables-alerter-879wd\" (UID: \"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90\") " pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851692    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-sys\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851714    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-lib-modules\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851745    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-var-lib-kubelet\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851791    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-tuned\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851821    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-run-systemd\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.851962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851847    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb296f1f-d101-4bd7-a981-3803bc01c51a-konnectivity-ca\") pod \"konnectivity-agent-n2d2h\" (UID: \"fb296f1f-d101-4bd7-a981-3803bc01c51a\") " pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851869    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851888    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzp8\" (UniqueName: \"kubernetes.io/projected/af6ff664-f55f-4650-b65a-3816ace0d8c0-kube-api-access-bqzp8\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851908    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aec6f6e8-33f0-43d3-b580-0b98f54b3bf3-hosts-file\") pod \"node-resolver-hp2v9\" (UID: \"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3\") " pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851931    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-cni-netd\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851953    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-registration-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851974    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-systemd\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.851996    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-run\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.852018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-systemd-units\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.852058    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.852089    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-858tc\" (UniqueName: \"kubernetes.io/projected/84437c7c-6768-4bbe-8e94-474b7d21a07b-kube-api-access-858tc\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.852115    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-sysctl-conf\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.852167    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2zw\" (UniqueName: \"kubernetes.io/projected/fdef210b-a23e-438e-a7d9-7978931d243e-kube-api-access-8w2zw\") pod \"node-ca-v2shg\" (UID: \"fdef210b-a23e-438e-a7d9-7978931d243e\") " pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.852193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.852668 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.852229    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-modprobe-d\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.880561 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.880531    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:32 +0000 UTC" deadline="2028-02-24 01:13:10.37048975 +0000 UTC"
May 25 11:17:33.880561 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.880561    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15349h55m36.489931903s"
May 25 11:17:33.916358 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.916339    2556 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:33.938497 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.938457    2556 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world"
May 25 11:17:33.953280 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953258    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-os-release\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.953380 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953293    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-var-lib-openvswitch\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.953380 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953310    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-run-ovn-kubernetes\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.953380 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953330    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-sys-fs\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.953380 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953349    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9607124d-82db-4c28-a3a4-5a49d4603955-ovnkube-config\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.953380 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953368    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-kubernetes\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.953656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953381    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-var-lib-openvswitch\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.953656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953386    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-run-ovn-kubernetes\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.953656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953386    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec6f6e8-33f0-43d3-b580-0b98f54b3bf3-tmp-dir\") pod \"node-resolver-hp2v9\" (UID: \"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3\") " pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.953656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953417    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-sys-fs\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.953656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953439    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13fe7228-2e31-41ea-866c-b1c045e78460-cni-binary-copy\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:33.953656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953452    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-kubernetes\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.953656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953468    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-cnibin\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.953656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953523    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-etc-kubernetes\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.953656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953552    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9607124d-82db-4c28-a3a4-5a49d4603955-ovnkube-script-lib\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953695    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90-iptables-alerter-script\") pod \"iptables-alerter-879wd\" (UID: \"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90\") " pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953710    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aec6f6e8-33f0-43d3-b580-0b98f54b3bf3-tmp-dir\") pod \"node-resolver-hp2v9\" (UID: \"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3\") " pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953728    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2l9\" (UniqueName: \"kubernetes.io/projected/37262c21-a42a-4ff7-90c7-bbe50a05d68b-kube-api-access-jd2l9\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953753    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-run-netns\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953780    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-etc-openvswitch\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953825    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-sysconfig\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953853    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9607124d-82db-4c28-a3a4-5a49d4603955-env-overrides\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953880    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-socket-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953895    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-etc-openvswitch\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953902    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af6ff664-f55f-4650-b65a-3816ace0d8c0-tmp\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-cnibin\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953979    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9607124d-82db-4c28-a3a4-5a49d4603955-ovnkube-config\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.953995    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-daemon-config\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954150    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90-iptables-alerter-script\") pod \"iptables-alerter-879wd\" (UID: \"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90\") " pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954175    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954252    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-socket-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954291    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-sysconfig\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954362    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9607124d-82db-4c28-a3a4-5a49d4603955-env-overrides\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954406    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-cni-bin\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954509    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-run-netns\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954051    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-cni-bin\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954556    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-c447b\" (UniqueName: \"kubernetes.io/projected/9607124d-82db-4c28-a3a4-5a49d4603955-kube-api-access-c447b\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954583    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954605    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdef210b-a23e-438e-a7d9-7978931d243e-host\") pod \"node-ca-v2shg\" (UID: \"fdef210b-a23e-438e-a7d9-7978931d243e\") " pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.954648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954642    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-os-release\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xmx\" (UniqueName: \"kubernetes.io/projected/13fe7228-2e31-41ea-866c-b1c045e78460-kube-api-access-l6xmx\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954698    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-kubelet\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954722    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb296f1f-d101-4bd7-a981-3803bc01c51a-agent-certs\") pod \"konnectivity-agent-n2d2h\" (UID: \"fb296f1f-d101-4bd7-a981-3803bc01c51a\") " pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954737    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9607124d-82db-4c28-a3a4-5a49d4603955-ovnkube-script-lib\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954745    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-device-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954777    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-b5vsx\" (UniqueName: \"kubernetes.io/projected/9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90-kube-api-access-b5vsx\") pod \"iptables-alerter-879wd\" (UID: \"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90\") " pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954801    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-sysctl-d\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954806    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fdef210b-a23e-438e-a7d9-7978931d243e-host\") pod \"node-ca-v2shg\" (UID: \"fdef210b-a23e-438e-a7d9-7978931d243e\") " pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954811    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-kubelet\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954839    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954867    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vmblr\" (UniqueName: \"kubernetes.io/projected/aec6f6e8-33f0-43d3-b580-0b98f54b3bf3-kube-api-access-vmblr\") pod \"node-resolver-hp2v9\" (UID: \"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3\") " pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954870    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-device-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954891    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-log-socket\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954917    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-host\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954936    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-etc-selinux\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-run-k8s-cni-cncf-io\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.955020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954968    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-run-netns\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954985    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-log-socket\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.954991    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-var-lib-cni-multus\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955030    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-run-ovn\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955088    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-sysctl-d\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955134    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-run-ovn\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955136    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9607124d-82db-4c28-a3a4-5a49d4603955-ovn-node-metrics-cert\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955138    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-host\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955267    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-system-cni-dir\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955299    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-var-lib-cni-bin\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955319    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-var-lib-kubelet\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-296xf\" (UniqueName: \"kubernetes.io/projected/bb98fe05-e021-483f-baf0-1e56cc1ad554-kube-api-access-296xf\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955369    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fdef210b-a23e-438e-a7d9-7978931d243e-serviceca\") pod \"node-ca-v2shg\" (UID: \"fdef210b-a23e-438e-a7d9-7978931d243e\") " pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955398    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-slash\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955423    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-run-openvswitch\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955448    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90-host-slash\") pod \"iptables-alerter-879wd\" (UID: \"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90\") " pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955472    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-sys\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955501    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-slash\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.955764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955520    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-lib-modules\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955548    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-var-lib-kubelet\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955565    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-tuned\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955571    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-run-openvswitch\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955587    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-run-systemd\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955603    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb296f1f-d101-4bd7-a981-3803bc01c51a-konnectivity-ca\") pod \"konnectivity-agent-n2d2h\" (UID: \"fb296f1f-d101-4bd7-a981-3803bc01c51a\") " pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955619    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90-host-slash\") pod \"iptables-alerter-879wd\" (UID: \"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90\") " pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955625    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955642    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzp8\" (UniqueName: \"kubernetes.io/projected/af6ff664-f55f-4650-b65a-3816ace0d8c0-kube-api-access-bqzp8\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955663    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aec6f6e8-33f0-43d3-b580-0b98f54b3bf3-hosts-file\") pod \"node-resolver-hp2v9\" (UID: \"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3\") " pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955670    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-sys\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955681    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955698    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13fe7228-2e31-41ea-866c-b1c045e78460-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-socket-dir-parent\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955716    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-run-systemd\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955738    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-cni-netd\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955755    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-registration-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.956579 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955771    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-systemd\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955786    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-run\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955800    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb98fe05-e021-483f-baf0-1e56cc1ad554-cni-binary-copy\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955807    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-lib-modules\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955856    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-var-lib-kubelet\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955866    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fdef210b-a23e-438e-a7d9-7978931d243e-serviceca\") pod \"node-ca-v2shg\" (UID: \"fdef210b-a23e-438e-a7d9-7978931d243e\") " pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955883    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-hostroot\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955909    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-systemd-units\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955948    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955973    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-858tc\" (UniqueName: \"kubernetes.io/projected/84437c7c-6768-4bbe-8e94-474b7d21a07b-kube-api-access-858tc\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.955998    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-sysctl-conf\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956022    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2zw\" (UniqueName: \"kubernetes.io/projected/fdef210b-a23e-438e-a7d9-7978931d243e-kube-api-access-8w2zw\") pod \"node-ca-v2shg\" (UID: \"fdef210b-a23e-438e-a7d9-7978931d243e\") " pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956047    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-conf-dir\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956071    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-run-multus-certs\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956095    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956119    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-modprobe-d\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956143    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-system-cni-dir\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:33.957278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956168    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-node-log\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956191    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13fe7228-2e31-41ea-866c-b1c045e78460-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956211    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-cni-dir\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956647    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb296f1f-d101-4bd7-a981-3803bc01c51a-konnectivity-ca\") pod \"konnectivity-agent-n2d2h\" (UID: \"fb296f1f-d101-4bd7-a981-3803bc01c51a\") " pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956792    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-systemd-units\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.956857    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:33.957034    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:33.957123    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs podName:37262c21-a42a-4ff7-90c7-bbe50a05d68b nodeName:}" failed. No retries permitted until 2026-05-25 11:17:34.457088873 +0000 UTC m=+3.087249735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs") pod "network-metrics-daemon-8w9qg" (UID: "37262c21-a42a-4ff7-90c7-bbe50a05d68b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.957608    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/aec6f6e8-33f0-43d3-b580-0b98f54b3bf3-hosts-file\") pod \"node-resolver-hp2v9\" (UID: \"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3\") " pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.957718    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-host-cni-netd\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.957782    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-registration-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.957823    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-systemd\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.958026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.957855    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-run\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.958564 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.958310    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb296f1f-d101-4bd7-a981-3803bc01c51a-agent-certs\") pod \"konnectivity-agent-n2d2h\" (UID: \"fb296f1f-d101-4bd7-a981-3803bc01c51a\") " pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:33.958564 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.958446    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-sysctl-conf\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.958564 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.958537    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-modprobe-d\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.958687 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.958591    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9607124d-82db-4c28-a3a4-5a49d4603955-node-log\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.960541 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.959192    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/84437c7c-6768-4bbe-8e94-474b7d21a07b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.960541 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.959335    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/af6ff664-f55f-4650-b65a-3816ace0d8c0-etc-tuned\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.960962 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.960799    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/af6ff664-f55f-4650-b65a-3816ace0d8c0-tmp\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:33.961751 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:33.961729    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:33.961841 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:33.961754    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:33.961841 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:33.961769    2556 projected.go:194] Error preparing data for projected volume kube-api-access-82md6 for pod openshift-network-diagnostics/network-check-target-2j6dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:33.961841 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:33.961833    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6 podName:4ed96c55-4683-4498-94f8-a8865cd52373 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:34.46181564 +0000 UTC m=+3.091976448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-82md6" (UniqueName: "kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6") pod "network-check-target-2j6dr" (UID: "4ed96c55-4683-4498-94f8-a8865cd52373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:33.962933 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.962893    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9607124d-82db-4c28-a3a4-5a49d4603955-ovn-node-metrics-cert\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.963286 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.963261    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c447b\" (UniqueName: \"kubernetes.io/projected/9607124d-82db-4c28-a3a4-5a49d4603955-kube-api-access-c447b\") pod \"ovnkube-node-794p7\" (UID: \"9607124d-82db-4c28-a3a4-5a49d4603955\") " pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:33.964337 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.964315    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5vsx\" (UniqueName: \"kubernetes.io/projected/9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90-kube-api-access-b5vsx\") pod \"iptables-alerter-879wd\" (UID: \"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90\") " pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:33.964778 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.964760    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2l9\" (UniqueName: \"kubernetes.io/projected/37262c21-a42a-4ff7-90c7-bbe50a05d68b-kube-api-access-jd2l9\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:33.964881 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.964801    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmblr\" (UniqueName: \"kubernetes.io/projected/aec6f6e8-33f0-43d3-b580-0b98f54b3bf3-kube-api-access-vmblr\") pod \"node-resolver-hp2v9\" (UID: \"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3\") " pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:33.965825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.965794    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2zw\" (UniqueName: \"kubernetes.io/projected/fdef210b-a23e-438e-a7d9-7978931d243e-kube-api-access-8w2zw\") pod \"node-ca-v2shg\" (UID: \"fdef210b-a23e-438e-a7d9-7978931d243e\") " pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:33.971615 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.971591    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-858tc\" (UniqueName: \"kubernetes.io/projected/84437c7c-6768-4bbe-8e94-474b7d21a07b-kube-api-access-858tc\") pod \"aws-ebs-csi-driver-node-9ddvs\" (UID: \"84437c7c-6768-4bbe-8e94-474b7d21a07b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:33.972312 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:33.972290    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzp8\" (UniqueName: \"kubernetes.io/projected/af6ff664-f55f-4650-b65a-3816ace0d8c0-kube-api-access-bqzp8\") pod \"tuned-9sv9g\" (UID: \"af6ff664-f55f-4650-b65a-3816ace0d8c0\") " pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:34.056514 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056471    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.056665 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056527    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13fe7228-2e31-41ea-866c-b1c045e78460-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.056665 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056554    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-socket-dir-parent\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.056665 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056581    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb98fe05-e021-483f-baf0-1e56cc1ad554-cni-binary-copy\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.056665 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056607    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-hostroot\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.056665 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056635    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-conf-dir\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.056665 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056635    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-socket-dir-parent\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.056665 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056657    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.056665 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056660    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-run-multus-certs\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056702    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-hostroot\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056713    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-system-cni-dir\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056738    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-conf-dir\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056745    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13fe7228-2e31-41ea-866c-b1c045e78460-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056701    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-run-multus-certs\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056746    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-system-cni-dir\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056775    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-cni-dir\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056799    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-os-release\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056830    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13fe7228-2e31-41ea-866c-b1c045e78460-cni-binary-copy\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056855    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-cnibin\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056856    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-cni-dir\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056890    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-etc-kubernetes\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056915    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-os-release\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056928    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-cnibin\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056964    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-daemon-config\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056979    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-cnibin\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.056997    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-os-release\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057016    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-etc-kubernetes\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057025    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xmx\" (UniqueName: \"kubernetes.io/projected/13fe7228-2e31-41ea-866c-b1c045e78460-kube-api-access-l6xmx\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057073    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-run-k8s-cni-cncf-io\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057079    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/13fe7228-2e31-41ea-866c-b1c045e78460-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057099    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-run-netns\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057125    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-var-lib-cni-multus\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057155    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-cnibin\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057174    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-run-netns\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057201    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-system-cni-dir\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057221    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bb98fe05-e021-483f-baf0-1e56cc1ad554-cni-binary-copy\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057227    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-var-lib-cni-bin\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057249    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/13fe7228-2e31-41ea-866c-b1c045e78460-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057264    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-var-lib-cni-bin\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057276    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-var-lib-kubelet\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057297    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-var-lib-cni-multus\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057304    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-296xf\" (UniqueName: \"kubernetes.io/projected/bb98fe05-e021-483f-baf0-1e56cc1ad554-kube-api-access-296xf\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057313    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/13fe7228-2e31-41ea-866c-b1c045e78460-os-release\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.057825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057327    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/13fe7228-2e31-41ea-866c-b1c045e78460-cni-binary-copy\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.058381 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057349    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-var-lib-kubelet\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.058381 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057350    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-system-cni-dir\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.058381 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057456    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bb98fe05-e021-483f-baf0-1e56cc1ad554-multus-daemon-config\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.058381 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.057472    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bb98fe05-e021-483f-baf0-1e56cc1ad554-host-run-k8s-cni-cncf-io\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.064881 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.064835    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-296xf\" (UniqueName: \"kubernetes.io/projected/bb98fe05-e021-483f-baf0-1e56cc1ad554-kube-api-access-296xf\") pod \"multus-hkkfj\" (UID: \"bb98fe05-e021-483f-baf0-1e56cc1ad554\") " pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.064881 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.064870    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xmx\" (UniqueName: \"kubernetes.io/projected/13fe7228-2e31-41ea-866c-b1c045e78460-kube-api-access-l6xmx\") pod \"multus-additional-cni-plugins-sh48r\" (UID: \"13fe7228-2e31-41ea-866c-b1c045e78460\") " pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.088040 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.088018    2556 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
May 25 11:17:34.137243 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.137216    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:34.146057 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.146037    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:34.152638 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.152616    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs"
May 25 11:17:34.158168 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.158142    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v2shg"
May 25 11:17:34.165657 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.165642    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-879wd"
May 25 11:17:34.172189 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.172170    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9sv9g"
May 25 11:17:34.178721 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.178697    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hp2v9"
May 25 11:17:34.186240 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.186226    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sh48r"
May 25 11:17:34.191802 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.191785    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-hkkfj"
May 25 11:17:34.460302 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.460229    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:34.460426 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:34.460326    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:34.460426 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:34.460383    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs podName:37262c21-a42a-4ff7-90c7-bbe50a05d68b nodeName:}" failed. No retries permitted until 2026-05-25 11:17:35.460361943 +0000 UTC m=+4.090522731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs") pod "network-metrics-daemon-8w9qg" (UID: "37262c21-a42a-4ff7-90c7-bbe50a05d68b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:34.489757 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:34.489728    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaec6f6e8_33f0_43d3_b580_0b98f54b3bf3.slice/crio-ea64ce3537809652566d082f5d89056d7614dd072ea807995c7dd036b25ed1d0 WatchSource:0}: Error finding container ea64ce3537809652566d082f5d89056d7614dd072ea807995c7dd036b25ed1d0: Status 404 returned error can't find the container with id ea64ce3537809652566d082f5d89056d7614dd072ea807995c7dd036b25ed1d0
May 25 11:17:34.491650 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:34.491610    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13fe7228_2e31_41ea_866c_b1c045e78460.slice/crio-641feeb3bd763369bcb45df75dc1f3915e708743e0f09bce87963db8fa14598d WatchSource:0}: Error finding container 641feeb3bd763369bcb45df75dc1f3915e708743e0f09bce87963db8fa14598d: Status 404 returned error can't find the container with id 641feeb3bd763369bcb45df75dc1f3915e708743e0f09bce87963db8fa14598d
May 25 11:17:34.494455 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:34.494431    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdef210b_a23e_438e_a7d9_7978931d243e.slice/crio-3302b1f97f6aa66080b28bc43f97c562780e4354cf1911cb155c793a92692499 WatchSource:0}: Error finding container 3302b1f97f6aa66080b28bc43f97c562780e4354cf1911cb155c793a92692499: Status 404 returned error can't find the container with id 3302b1f97f6aa66080b28bc43f97c562780e4354cf1911cb155c793a92692499
May 25 11:17:34.495212 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:34.495183    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d72f58b_2d2b_49e1_9d22_42b6fa8d8f90.slice/crio-efe46e2a5b5bee95a830d035741ba96b2a03b9cb83a18b07f1c5f2afbd5280d5 WatchSource:0}: Error finding container efe46e2a5b5bee95a830d035741ba96b2a03b9cb83a18b07f1c5f2afbd5280d5: Status 404 returned error can't find the container with id efe46e2a5b5bee95a830d035741ba96b2a03b9cb83a18b07f1c5f2afbd5280d5
May 25 11:17:34.496090 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:34.496071    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb98fe05_e021_483f_baf0_1e56cc1ad554.slice/crio-de311b5a75026e3f2ecb3f0377794546a935a2e8de08339871e7ed5af0605699 WatchSource:0}: Error finding container de311b5a75026e3f2ecb3f0377794546a935a2e8de08339871e7ed5af0605699: Status 404 returned error can't find the container with id de311b5a75026e3f2ecb3f0377794546a935a2e8de08339871e7ed5af0605699
May 25 11:17:34.496998 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:17:34.496828    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb296f1f_d101_4bd7_a981_3803bc01c51a.slice/crio-2bf2a3a4bc56117720ebcef5ae91b57b5dcb7f635ab67c3d187609f1217584b4 WatchSource:0}: Error finding container 2bf2a3a4bc56117720ebcef5ae91b57b5dcb7f635ab67c3d187609f1217584b4: Status 404 returned error can't find the container with id 2bf2a3a4bc56117720ebcef5ae91b57b5dcb7f635ab67c3d187609f1217584b4
May 25 11:17:34.561171 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.561048    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:34.561275 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:34.561189    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:34.561275 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:34.561207    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:34.561275 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:34.561216    2556 projected.go:194] Error preparing data for projected volume kube-api-access-82md6 for pod openshift-network-diagnostics/network-check-target-2j6dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:34.561275 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:34.561260    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6 podName:4ed96c55-4683-4498-94f8-a8865cd52373 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:35.561246089 +0000 UTC m=+4.191406882 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-82md6" (UniqueName: "kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6") pod "network-check-target-2j6dr" (UID: "4ed96c55-4683-4498-94f8-a8865cd52373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:34.881104 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.881039    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-24 11:12:32 +0000 UTC" deadline="2027-11-15 00:54:56.477082356 +0000 UTC"
May 25 11:17:34.881104 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.881074    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12925h37m21.596011798s"
May 25 11:17:34.977270 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.976794    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:34.977270 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:34.976905    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:34.990422 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.989792    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal" event={"ID":"b3204d4e276a6036a6def62b77dcf7af","Type":"ContainerStarted","Data":"a1329e5a3227b1c5209257934ceb3a958c62bd6f0f3cbe591144f8d54a2a2587"}
May 25 11:17:34.994852 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.994794    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerStarted","Data":"442a0a5f886de6fc4feb47ad567a3a6382336d4e99686001ee6cd3c829a8c172"}
May 25 11:17:34.999594 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:34.999568    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs" event={"ID":"84437c7c-6768-4bbe-8e94-474b7d21a07b","Type":"ContainerStarted","Data":"d02fe13f0440f099e628d07c40e272780d232d7e6f769a410f5ac8d99f3b9fb8"}
May 25 11:17:35.008814 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.008646    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hkkfj" event={"ID":"bb98fe05-e021-483f-baf0-1e56cc1ad554","Type":"ContainerStarted","Data":"de311b5a75026e3f2ecb3f0377794546a935a2e8de08339871e7ed5af0605699"}
May 25 11:17:35.012049 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.012009    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-879wd" event={"ID":"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90","Type":"ContainerStarted","Data":"efe46e2a5b5bee95a830d035741ba96b2a03b9cb83a18b07f1c5f2afbd5280d5"}
May 25 11:17:35.014209 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.014163    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v2shg" event={"ID":"fdef210b-a23e-438e-a7d9-7978931d243e","Type":"ContainerStarted","Data":"3302b1f97f6aa66080b28bc43f97c562780e4354cf1911cb155c793a92692499"}
May 25 11:17:35.018022 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.017981    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sh48r" event={"ID":"13fe7228-2e31-41ea-866c-b1c045e78460","Type":"ContainerStarted","Data":"641feeb3bd763369bcb45df75dc1f3915e708743e0f09bce87963db8fa14598d"}
May 25 11:17:35.020161 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.020112    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hp2v9" event={"ID":"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3","Type":"ContainerStarted","Data":"ea64ce3537809652566d082f5d89056d7614dd072ea807995c7dd036b25ed1d0"}
May 25 11:17:35.022958 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.022506    2556 generic.go:358] "Generic (PLEG): container finished" podID="8b79a830b6f70433eb778f4f12409255" containerID="64ae7f9775d955737d826976c17a4df1f0468e92cf76d4a9c509dce396fd6bcc" exitCode=0
May 25 11:17:35.022958 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.022573    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" event={"ID":"8b79a830b6f70433eb778f4f12409255","Type":"ContainerDied","Data":"64ae7f9775d955737d826976c17a4df1f0468e92cf76d4a9c509dce396fd6bcc"}
May 25 11:17:35.024623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.024572    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9sv9g" event={"ID":"af6ff664-f55f-4650-b65a-3816ace0d8c0","Type":"ContainerStarted","Data":"21056c00b160ae427510ee56a7a2a98febfe35a3038c13f9eaf30b02d1cb4bd3"}
May 25 11:17:35.025930 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.025891    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n2d2h" event={"ID":"fb296f1f-d101-4bd7-a981-3803bc01c51a","Type":"ContainerStarted","Data":"2bf2a3a4bc56117720ebcef5ae91b57b5dcb7f635ab67c3d187609f1217584b4"}
May 25 11:17:35.041634 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.040571    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-161.ec2.internal" podStartSLOduration=2.040556638 podStartE2EDuration="2.040556638s" podCreationTimestamp="2026-05-25 11:17:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:35.011266278 +0000 UTC m=+3.641427091" watchObservedRunningTime="2026-05-25 11:17:35.040556638 +0000 UTC m=+3.670717449"
May 25 11:17:35.467259 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.467216    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:35.467439 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:35.467356    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:35.467439 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:35.467416    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs podName:37262c21-a42a-4ff7-90c7-bbe50a05d68b nodeName:}" failed. No retries permitted until 2026-05-25 11:17:37.46739481 +0000 UTC m=+6.097555611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs") pod "network-metrics-daemon-8w9qg" (UID: "37262c21-a42a-4ff7-90c7-bbe50a05d68b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:35.567800 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.567725    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:35.568050 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:35.567868    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:35.568050 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:35.567891    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:35.568050 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:35.567903    2556 projected.go:194] Error preparing data for projected volume kube-api-access-82md6 for pod openshift-network-diagnostics/network-check-target-2j6dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:35.568050 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:35.567963    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6 podName:4ed96c55-4683-4498-94f8-a8865cd52373 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:37.567943905 +0000 UTC m=+6.198104707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-82md6" (UniqueName: "kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6") pod "network-check-target-2j6dr" (UID: "4ed96c55-4683-4498-94f8-a8865cd52373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:35.978999 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:35.978442    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:35.978999 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:35.978600    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:36.040813 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:36.040753    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" event={"ID":"8b79a830b6f70433eb778f4f12409255","Type":"ContainerStarted","Data":"fdee3b2b1bc6e9111e602f3e9a77dcc1daf8ed11ba735deffb9cc37f989b0213"}
May 25 11:17:36.057310 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:36.057147    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-161.ec2.internal" podStartSLOduration=3.057129745 podStartE2EDuration="3.057129745s" podCreationTimestamp="2026-05-25 11:17:33 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:17:36.05639546 +0000 UTC m=+4.686556274" watchObservedRunningTime="2026-05-25 11:17:36.057129745 +0000 UTC m=+4.687290556"
May 25 11:17:36.977292 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:36.976791    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:36.977292 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:36.976911    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:37.482340 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:37.482297    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:37.482807 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:37.482434    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:37.482807 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:37.482516    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs podName:37262c21-a42a-4ff7-90c7-bbe50a05d68b nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.482496171 +0000 UTC m=+10.112656975 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs") pod "network-metrics-daemon-8w9qg" (UID: "37262c21-a42a-4ff7-90c7-bbe50a05d68b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:37.583739 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:37.583460    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:37.583739 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:37.583714    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:37.583739 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:37.583736    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:37.584006 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:37.583751    2556 projected.go:194] Error preparing data for projected volume kube-api-access-82md6 for pod openshift-network-diagnostics/network-check-target-2j6dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:37.584006 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:37.583814    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6 podName:4ed96c55-4683-4498-94f8-a8865cd52373 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:41.583796435 +0000 UTC m=+10.213957228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-82md6" (UniqueName: "kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6") pod "network-check-target-2j6dr" (UID: "4ed96c55-4683-4498-94f8-a8865cd52373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:37.977761 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:37.977684    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:37.977906 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:37.977829    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:38.976392 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:38.976360    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:38.976858 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:38.976500    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:39.976413 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:39.976292    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:39.976839 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:39.976451    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:40.976646 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:40.976611    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:40.977085 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:40.976736    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:41.517059 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:41.516964    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:41.517249 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:41.517126    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:41.517249 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:41.517199    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs podName:37262c21-a42a-4ff7-90c7-bbe50a05d68b nodeName:}" failed. No retries permitted until 2026-05-25 11:17:49.517179546 +0000 UTC m=+18.147340349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs") pod "network-metrics-daemon-8w9qg" (UID: "37262c21-a42a-4ff7-90c7-bbe50a05d68b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:41.617450 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:41.617374    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:41.617650 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:41.617553    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:41.617650 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:41.617572    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:41.617650 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:41.617584    2556 projected.go:194] Error preparing data for projected volume kube-api-access-82md6 for pod openshift-network-diagnostics/network-check-target-2j6dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:41.617650 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:41.617640    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6 podName:4ed96c55-4683-4498-94f8-a8865cd52373 nodeName:}" failed. No retries permitted until 2026-05-25 11:17:49.617622087 +0000 UTC m=+18.247782874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-82md6" (UniqueName: "kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6") pod "network-check-target-2j6dr" (UID: "4ed96c55-4683-4498-94f8-a8865cd52373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:41.977551 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:41.977376    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:41.977551 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:41.977505    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:42.976280 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:42.976246    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:42.976446 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:42.976368    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:43.976782 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:43.976747    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:43.977249 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:43.976860    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:44.976253 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:44.976218    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:44.976435 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:44.976330    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:45.976612 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:45.976574    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:45.977386 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:45.976704    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:46.975822 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:46.975791    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:46.975987 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:46.975904    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:47.976389 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:47.976356    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:47.976845 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:47.976513    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:48.976382 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:48.976331    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:48.976540 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:48.976471    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:49.577392 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:49.577356    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:49.577596 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:49.577503    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:49.577596 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:49.577570    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs podName:37262c21-a42a-4ff7-90c7-bbe50a05d68b nodeName:}" failed. No retries permitted until 2026-05-25 11:18:05.57755373 +0000 UTC m=+34.207714532 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs") pod "network-metrics-daemon-8w9qg" (UID: "37262c21-a42a-4ff7-90c7-bbe50a05d68b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:17:49.678058 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:49.678020    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:49.678229 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:49.678207    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:17:49.678294 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:49.678234    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:17:49.678294 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:49.678245    2556 projected.go:194] Error preparing data for projected volume kube-api-access-82md6 for pod openshift-network-diagnostics/network-check-target-2j6dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:49.678380 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:49.678300    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6 podName:4ed96c55-4683-4498-94f8-a8865cd52373 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:05.678281828 +0000 UTC m=+34.308442622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-82md6" (UniqueName: "kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6") pod "network-check-target-2j6dr" (UID: "4ed96c55-4683-4498-94f8-a8865cd52373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:17:49.976592 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:49.976526    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:49.977001 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:49.976664    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:50.976157 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:50.976130    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:50.976318 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:50.976232    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:51.977227 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:51.977197    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:51.977597 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:51.977284    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:52.976182 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:52.976027    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:52.976274 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:52.976257    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:53.070949 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.070779    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:17:53.071589 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.071159    2556 generic.go:358] "Generic (PLEG): container finished" podID="9607124d-82db-4c28-a3a4-5a49d4603955" containerID="c7dcbf1f9449981b05d792d9976b38f17b0d43323a5a545a8b77e1512128a519" exitCode=1
May 25 11:17:53.071589 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.071223    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerDied","Data":"c7dcbf1f9449981b05d792d9976b38f17b0d43323a5a545a8b77e1512128a519"}
May 25 11:17:53.071589 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.071245    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerStarted","Data":"8aa2f3d7e2c5fb2013b1555f1522fde44faf5a14a9808cf5934fcc93476ab24a"}
May 25 11:17:53.072640 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.072614    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs" event={"ID":"84437c7c-6768-4bbe-8e94-474b7d21a07b","Type":"ContainerStarted","Data":"407a7c3ee10dd84bcca9d061ee3c7a14a5599e9733b914758069a81f38ad8187"}
May 25 11:17:53.073917 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.073892    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-hkkfj" event={"ID":"bb98fe05-e021-483f-baf0-1e56cc1ad554","Type":"ContainerStarted","Data":"bd8114a6ab74a6a37545a5d3674d21dbed4a7d1c42e37df278bdb71b1c340ffa"}
May 25 11:17:53.075866 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.075835    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v2shg" event={"ID":"fdef210b-a23e-438e-a7d9-7978931d243e","Type":"ContainerStarted","Data":"bdf6b9ec9b6a64feff30fddf87e219675312bee91b73bad94c2ddc1720869e30"}
May 25 11:17:53.077295 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.077272    2556 generic.go:358] "Generic (PLEG): container finished" podID="13fe7228-2e31-41ea-866c-b1c045e78460" containerID="dab057e3d23a5e8ecd92adba8ef5cf4426a462916b63fd164cba75490372ba49" exitCode=0
May 25 11:17:53.077417 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.077341    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sh48r" event={"ID":"13fe7228-2e31-41ea-866c-b1c045e78460","Type":"ContainerDied","Data":"dab057e3d23a5e8ecd92adba8ef5cf4426a462916b63fd164cba75490372ba49"}
May 25 11:17:53.078798 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.078780    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hp2v9" event={"ID":"aec6f6e8-33f0-43d3-b580-0b98f54b3bf3","Type":"ContainerStarted","Data":"d433903402ab47fa252d4918ed8c9304a85cd3212adadc5bacba0c556249ba70"}
May 25 11:17:53.080170 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.080150    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9sv9g" event={"ID":"af6ff664-f55f-4650-b65a-3816ace0d8c0","Type":"ContainerStarted","Data":"70442fdadf9701a399e3903724464890f9f852f532cee43881739cf86dfbccbc"}
May 25 11:17:53.081473 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.081452    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n2d2h" event={"ID":"fb296f1f-d101-4bd7-a981-3803bc01c51a","Type":"ContainerStarted","Data":"8cb12a122bf2db62fa2941b7b8ff7c44ea63c70a671159025d3782f6eb5f1265"}
May 25 11:17:53.103272 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.103231    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-hkkfj" podStartSLOduration=3.020125157 podStartE2EDuration="21.103221271s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:17:34.498641238 +0000 UTC m=+3.128802028" lastFinishedPulling="2026-05-25 11:17:52.581737342 +0000 UTC m=+21.211898142" observedRunningTime="2026-05-25 11:17:53.102175872 +0000 UTC m=+21.732336679" watchObservedRunningTime="2026-05-25 11:17:53.103221271 +0000 UTC m=+21.733382080"
May 25 11:17:53.118377 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.118339    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v2shg" podStartSLOduration=3.187791556 podStartE2EDuration="21.118327008s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:17:34.497297214 +0000 UTC m=+3.127458003" lastFinishedPulling="2026-05-25 11:17:52.427832665 +0000 UTC m=+21.057993455" observedRunningTime="2026-05-25 11:17:53.118266502 +0000 UTC m=+21.748427311" watchObservedRunningTime="2026-05-25 11:17:53.118327008 +0000 UTC m=+21.748487817"
May 25 11:17:53.157898 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.157855    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9sv9g" podStartSLOduration=3.121754375 podStartE2EDuration="21.15784058s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:17:34.505581075 +0000 UTC m=+3.135741862" lastFinishedPulling="2026-05-25 11:17:52.541667276 +0000 UTC m=+21.171828067" observedRunningTime="2026-05-25 11:17:53.138292912 +0000 UTC m=+21.768453722" watchObservedRunningTime="2026-05-25 11:17:53.15784058 +0000 UTC m=+21.788001392"
May 25 11:17:53.158011 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.157986    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-hp2v9" podStartSLOduration=3.109938056 podStartE2EDuration="21.1579796s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:17:34.49199571 +0000 UTC m=+3.122156503" lastFinishedPulling="2026-05-25 11:17:52.540037244 +0000 UTC m=+21.170198047" observedRunningTime="2026-05-25 11:17:53.157433802 +0000 UTC m=+21.787594613" watchObservedRunningTime="2026-05-25 11:17:53.1579796 +0000 UTC m=+21.788140411"
May 25 11:17:53.194768 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.194732    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n2d2h" podStartSLOduration=3.266480033 podStartE2EDuration="21.194720425s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:17:34.49959084 +0000 UTC m=+3.129751628" lastFinishedPulling="2026-05-25 11:17:52.427831215 +0000 UTC m=+21.057992020" observedRunningTime="2026-05-25 11:17:53.194419905 +0000 UTC m=+21.824580719" watchObservedRunningTime="2026-05-25 11:17:53.194720425 +0000 UTC m=+21.824881235"
May 25 11:17:53.891534 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.891475    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
May 25 11:17:53.908836 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.908749    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-25T11:17:53.891518318Z","UUID":"d9fb267f-1659-4866-a913-dc9b0485eab7","Handler":null,"Name":"","Endpoint":""}
May 25 11:17:53.910820 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.910773    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
May 25 11:17:53.910820 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.910799    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
May 25 11:17:53.976142 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:53.976121    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:53.976275 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:53.976249    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:54.085760 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.085735    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:17:54.086229 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.086132    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerStarted","Data":"205dbdf8275777259e8579cb8cac33ef5a1796c646b0cec0201f7aa46b902338"}
May 25 11:17:54.086229 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.086178    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerStarted","Data":"75967458fd72c6628a21c0e5dd49def66aa31eee35c389b007797e3e2c477385"}
May 25 11:17:54.086229 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.086192    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerStarted","Data":"ffcbb1ffb535914ba1b467c92ebe545f253fa34f31c2f81f7e519fdd1413ab02"}
May 25 11:17:54.086229 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.086205    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerStarted","Data":"84bc96db643359e819a77e7d4f3241dfec406b663e5f7f1b0f516d71ef3f7886"}
May 25 11:17:54.088014 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.087993    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs" event={"ID":"84437c7c-6768-4bbe-8e94-474b7d21a07b","Type":"ContainerStarted","Data":"749e31e46d4335f9d3e518dc53481ba7ca330c3be61c8b1bdafa714fadd167b1"}
May 25 11:17:54.089474 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.089417    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-879wd" event={"ID":"9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90","Type":"ContainerStarted","Data":"0184e7a8334cc2a2025364d8509239c717393101f4baf2cf22ff0a24b2deab14"}
May 25 11:17:54.127042 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.127003    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-879wd" podStartSLOduration=4.083452779 podStartE2EDuration="22.126989193s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:17:34.497367659 +0000 UTC m=+3.127528447" lastFinishedPulling="2026-05-25 11:17:52.540904068 +0000 UTC m=+21.171064861" observedRunningTime="2026-05-25 11:17:54.126674921 +0000 UTC m=+22.756835730" watchObservedRunningTime="2026-05-25 11:17:54.126989193 +0000 UTC m=+22.757150006"
May 25 11:17:54.465161 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.465083    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:54.465765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.465741    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:54.976713 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:54.976693    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:54.976841 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:54.976804    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:55.093200 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:55.093167    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs" event={"ID":"84437c7c-6768-4bbe-8e94-474b7d21a07b","Type":"ContainerStarted","Data":"4abb472c15167c86c6c4f8f3643f1bb68c42cad8e14bb4dc809c36e19fb54774"}
May 25 11:17:55.093614 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:55.093597    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:55.093938 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:55.093912    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n2d2h"
May 25 11:17:55.139023 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:55.138934    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9ddvs" podStartSLOduration=2.778172158 podStartE2EDuration="23.13892249s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:17:34.500681423 +0000 UTC m=+3.130842214" lastFinishedPulling="2026-05-25 11:17:54.861431754 +0000 UTC m=+23.491592546" observedRunningTime="2026-05-25 11:17:55.116800505 +0000 UTC m=+23.746961307" watchObservedRunningTime="2026-05-25 11:17:55.13892249 +0000 UTC m=+23.769083302"
May 25 11:17:55.976603 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:55.976424    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:55.976819 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:55.976699    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:56.098255 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:56.098224    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:17:56.098720 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:56.098649    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerStarted","Data":"65e9a39fa9d6f4b321dc9eb23a62b0dbdbed577c50d41cfba98c2107ad36ae72"}
May 25 11:17:56.976258 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:56.976226    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:56.976431 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:56.976351    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:57.976163 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:57.975936    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:57.977010 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:57.976172    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:17:58.105836 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:58.105815    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:17:58.106271 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:58.106247    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerStarted","Data":"895d452ca930fdc5b5885b5a661bb011e79af273c11745b1d11e286665a939b3"}
May 25 11:17:58.106546 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:58.106516    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:58.106546 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:58.106544    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:58.106772 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:58.106738    2556 scope.go:117] "RemoveContainer" containerID="c7dcbf1f9449981b05d792d9976b38f17b0d43323a5a545a8b77e1512128a519"
May 25 11:17:58.108547 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:58.108520    2556 generic.go:358] "Generic (PLEG): container finished" podID="13fe7228-2e31-41ea-866c-b1c045e78460" containerID="055d0255613c836ebe2c20bd3ef4f883480d69646257fee1699e0b8651a948c4" exitCode=0
May 25 11:17:58.108661 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:58.108564    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sh48r" event={"ID":"13fe7228-2e31-41ea-866c-b1c045e78460","Type":"ContainerDied","Data":"055d0255613c836ebe2c20bd3ef4f883480d69646257fee1699e0b8651a948c4"}
May 25 11:17:58.122763 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:58.122739    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:58.976255 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:58.976232    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:17:58.976536 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:58.976344    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:17:59.113245 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:59.113219    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:17:59.113536 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:59.113511    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" event={"ID":"9607124d-82db-4c28-a3a4-5a49d4603955","Type":"ContainerStarted","Data":"844ccb140b93bcd29fc1ef6fded4dfadf232967b7b34bb7a3f4ba76336c234dd"}
May 25 11:17:59.113838 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:59.113814    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:59.115443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:59.115423    2556 generic.go:358] "Generic (PLEG): container finished" podID="13fe7228-2e31-41ea-866c-b1c045e78460" containerID="08f17784d12c96819dd7d3ad1bb1852b232d5b1720187029eefcbc442b263c43" exitCode=0
May 25 11:17:59.115546 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:59.115465    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sh48r" event={"ID":"13fe7228-2e31-41ea-866c-b1c045e78460","Type":"ContainerDied","Data":"08f17784d12c96819dd7d3ad1bb1852b232d5b1720187029eefcbc442b263c43"}
May 25 11:17:59.128850 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:59.128832    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:17:59.183374 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:59.183294    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-794p7" podStartSLOduration=9.13074571 podStartE2EDuration="27.183279877s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:17:34.505210667 +0000 UTC m=+3.135371470" lastFinishedPulling="2026-05-25 11:17:52.557744846 +0000 UTC m=+21.187905637" observedRunningTime="2026-05-25 11:17:59.146410134 +0000 UTC m=+27.776570945" watchObservedRunningTime="2026-05-25 11:17:59.183279877 +0000 UTC m=+27.813440686"
May 25 11:17:59.976914 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:17:59.976848    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:17:59.977296 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:17:59.976969    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:18:00.119007 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:00.118980    2556 generic.go:358] "Generic (PLEG): container finished" podID="13fe7228-2e31-41ea-866c-b1c045e78460" containerID="2595dc66222f36eddc375af61c42710c7a0e4b754a8b7e004607a85dd7bd114c" exitCode=0
May 25 11:18:00.119116 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:00.119040    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sh48r" event={"ID":"13fe7228-2e31-41ea-866c-b1c045e78460","Type":"ContainerDied","Data":"2595dc66222f36eddc375af61c42710c7a0e4b754a8b7e004607a85dd7bd114c"}
May 25 11:18:00.976399 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:00.976372    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:00.976545 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:00.976467    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:18:01.977077 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:01.977041    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:01.977548 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:01.977155    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:18:02.975900 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:02.975862    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:02.976084 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:02.975986    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:18:03.976319 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:03.976277    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:03.976722 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:03.976420    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:18:04.976304 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:04.976274    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:04.976506 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:04.976390    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:18:05.594846 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:05.594816    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:05.595073 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:05.594911    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:18:05.595073 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:05.594955    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs podName:37262c21-a42a-4ff7-90c7-bbe50a05d68b nodeName:}" failed. No retries permitted until 2026-05-25 11:18:37.594942966 +0000 UTC m=+66.225103754 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs") pod "network-metrics-daemon-8w9qg" (UID: "37262c21-a42a-4ff7-90c7-bbe50a05d68b") : object "openshift-multus"/"metrics-daemon-secret" not registered
May 25 11:18:05.695548 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:05.695521    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:05.695684 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:05.695618    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
May 25 11:18:05.695684 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:05.695629    2556 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
May 25 11:18:05.695684 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:05.695637    2556 projected.go:194] Error preparing data for projected volume kube-api-access-82md6 for pod openshift-network-diagnostics/network-check-target-2j6dr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:18:05.695684 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:05.695678    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6 podName:4ed96c55-4683-4498-94f8-a8865cd52373 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:37.695666328 +0000 UTC m=+66.325827115 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-82md6" (UniqueName: "kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6") pod "network-check-target-2j6dr" (UID: "4ed96c55-4683-4498-94f8-a8865cd52373") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
May 25 11:18:05.976647 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:05.976621    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:05.976970 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:05.976727    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:18:06.130617 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:06.130587    2556 generic.go:358] "Generic (PLEG): container finished" podID="13fe7228-2e31-41ea-866c-b1c045e78460" containerID="a4c724b65b722ae336cbabd4e96f14e57e3cde99d30360d4cc00f152ef35d1d5" exitCode=0
May 25 11:18:06.130744 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:06.130649    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sh48r" event={"ID":"13fe7228-2e31-41ea-866c-b1c045e78460","Type":"ContainerDied","Data":"a4c724b65b722ae336cbabd4e96f14e57e3cde99d30360d4cc00f152ef35d1d5"}
May 25 11:18:06.976271 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:06.976244    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:06.976523 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:06.976325    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:18:07.135212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:07.135184    2556 generic.go:358] "Generic (PLEG): container finished" podID="13fe7228-2e31-41ea-866c-b1c045e78460" containerID="5cc4986a5dbd2b3f937814fa2022c687adbd02e9150de676982ae25969d3e791" exitCode=0
May 25 11:18:07.135531 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:07.135226    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sh48r" event={"ID":"13fe7228-2e31-41ea-866c-b1c045e78460","Type":"ContainerDied","Data":"5cc4986a5dbd2b3f937814fa2022c687adbd02e9150de676982ae25969d3e791"}
May 25 11:18:07.976740 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:07.976712    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:07.976908 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:07.976803    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:18:08.139796 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:08.139768    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sh48r" event={"ID":"13fe7228-2e31-41ea-866c-b1c045e78460","Type":"ContainerStarted","Data":"cf326539dcd426acfecc9359c895dda8a7e168548fecf98ab8c58c2d9d915de7"}
May 25 11:18:08.164638 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:08.164601    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sh48r" podStartSLOduration=4.939174733 podStartE2EDuration="36.164588955s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:17:34.493861411 +0000 UTC m=+3.124022202" lastFinishedPulling="2026-05-25 11:18:05.719275635 +0000 UTC m=+34.349436424" observedRunningTime="2026-05-25 11:18:08.164411645 +0000 UTC m=+36.794572454" watchObservedRunningTime="2026-05-25 11:18:08.164588955 +0000 UTC m=+36.794749765"
May 25 11:18:08.976347 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:08.976176    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:08.976501 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:08.976403    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:18:09.976133 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:09.976106    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:09.976498 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:09.976199    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:18:10.976060 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:10.976031    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:10.976211 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:10.976123    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:18:11.711959 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:11.711926    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2j6dr"]
May 25 11:18:11.712131 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:11.712051    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:11.712201 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:11.712184    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:18:11.715069 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:11.715044    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8w9qg"]
May 25 11:18:11.715174 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:11.715141    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:11.715307 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:11.715242    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:18:12.976780 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:12.976750    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:12.977386 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:12.976839    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:18:13.976154 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:13.976128    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:13.976305 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:13.976217    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8w9qg" podUID="37262c21-a42a-4ff7-90c7-bbe50a05d68b"
May 25 11:18:14.976227 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:14.976201    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:14.976657 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:14.976290    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2j6dr" podUID="4ed96c55-4683-4498-94f8-a8865cd52373"
May 25 11:18:15.682971 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.682950    2556 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-128-161.ec2.internal" event="NodeReady"
May 25 11:18:15.683089 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.683030    2556 kubelet_node_status.go:550] "Fast updating node status as it just became ready"
May 25 11:18:15.747299 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.747268    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pw56p"]
May 25 11:18:15.749110 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.749092    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.751171 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.751146    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ggmkh"]
May 25 11:18:15.751805 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.751786    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mgwng\""
May 25 11:18:15.751895 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.751796    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
May 25 11:18:15.751895 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.751820    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
May 25 11:18:15.752652 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.752639    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.755522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.755502    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
May 25 11:18:15.755522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.755515    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
May 25 11:18:15.756564 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.756159    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
May 25 11:18:15.756564 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.756220    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-92tcr\""
May 25 11:18:15.756960 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.756685    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
May 25 11:18:15.761797 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.761779    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pw56p"]
May 25 11:18:15.769708 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.769685    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ggmkh"]
May 25 11:18:15.846235 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.846213    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fgzh7"]
May 25 11:18:15.848056 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.848043    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fgzh7"
May 25 11:18:15.850740 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.850722    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
May 25 11:18:15.850859 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.850841    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4jjkv\""
May 25 11:18:15.851011 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.850995    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
May 25 11:18:15.851098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.851084    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
May 25 11:18:15.862281 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.862264    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fgzh7"]
May 25 11:18:15.869893 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.869874    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32313dcb-927f-4c70-917c-f876bfc59823-crio-socket\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.869983 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.869904    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32313dcb-927f-4c70-917c-f876bfc59823-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.869983 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.869930    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-metrics-tls\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.869983 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.869952    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-tmp-dir\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.869983 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.869976    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7phb\" (UniqueName: \"kubernetes.io/projected/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-kube-api-access-f7phb\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.870178 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.869992    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32313dcb-927f-4c70-917c-f876bfc59823-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.870178 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.870022    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-config-volume\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.870178 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.870134    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32313dcb-927f-4c70-917c-f876bfc59823-data-volume\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.870299 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.870192    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpjn7\" (UniqueName: \"kubernetes.io/projected/32313dcb-927f-4c70-917c-f876bfc59823-kube-api-access-wpjn7\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.971083 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971035    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-config-volume\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.971177 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971080    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32313dcb-927f-4c70-917c-f876bfc59823-data-volume\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.971177 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971114    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-wpjn7\" (UniqueName: \"kubernetes.io/projected/32313dcb-927f-4c70-917c-f876bfc59823-kube-api-access-wpjn7\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.971177 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971144    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwm24\" (UniqueName: \"kubernetes.io/projected/2f5aae4f-4930-47cc-be9f-2b971dbd12af-kube-api-access-gwm24\") pod \"ingress-canary-fgzh7\" (UID: \"2f5aae4f-4930-47cc-be9f-2b971dbd12af\") " pod="openshift-ingress-canary/ingress-canary-fgzh7"
May 25 11:18:15.971311 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971233    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32313dcb-927f-4c70-917c-f876bfc59823-crio-socket\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.971311 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971268    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32313dcb-927f-4c70-917c-f876bfc59823-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.971311 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971288    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-metrics-tls\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.971311 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971304    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-tmp-dir\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.971513 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971331    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f7phb\" (UniqueName: \"kubernetes.io/projected/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-kube-api-access-f7phb\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.971513 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971357    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f5aae4f-4930-47cc-be9f-2b971dbd12af-cert\") pod \"ingress-canary-fgzh7\" (UID: \"2f5aae4f-4930-47cc-be9f-2b971dbd12af\") " pod="openshift-ingress-canary/ingress-canary-fgzh7"
May 25 11:18:15.971513 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971384    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32313dcb-927f-4c70-917c-f876bfc59823-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.971513 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971424    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/32313dcb-927f-4c70-917c-f876bfc59823-data-volume\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.971678 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971601    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/32313dcb-927f-4c70-917c-f876bfc59823-crio-socket\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.971678 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971653    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-config-volume\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.971678 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971658    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-tmp-dir\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.971826 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.971807    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/32313dcb-927f-4c70-917c-f876bfc59823-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.975136 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.975015    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-metrics-tls\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.975205 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.975047    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/32313dcb-927f-4c70-917c-f876bfc59823-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:15.976069 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.976054    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:15.978530 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.978511    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:18:15.978879 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.978511    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78wx\""
May 25 11:18:15.979167 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.979146    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7phb\" (UniqueName: \"kubernetes.io/projected/7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4-kube-api-access-f7phb\") pod \"dns-default-pw56p\" (UID: \"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4\") " pod="openshift-dns/dns-default-pw56p"
May 25 11:18:15.979362 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:15.979341    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpjn7\" (UniqueName: \"kubernetes.io/projected/32313dcb-927f-4c70-917c-f876bfc59823-kube-api-access-wpjn7\") pod \"insights-runtime-extractor-ggmkh\" (UID: \"32313dcb-927f-4c70-917c-f876bfc59823\") " pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:16.060813 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.060792    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pw56p"
May 25 11:18:16.066341 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.066324    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ggmkh"
May 25 11:18:16.072046 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.072009    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gwm24\" (UniqueName: \"kubernetes.io/projected/2f5aae4f-4930-47cc-be9f-2b971dbd12af-kube-api-access-gwm24\") pod \"ingress-canary-fgzh7\" (UID: \"2f5aae4f-4930-47cc-be9f-2b971dbd12af\") " pod="openshift-ingress-canary/ingress-canary-fgzh7"
May 25 11:18:16.072138 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.072092    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f5aae4f-4930-47cc-be9f-2b971dbd12af-cert\") pod \"ingress-canary-fgzh7\" (UID: \"2f5aae4f-4930-47cc-be9f-2b971dbd12af\") " pod="openshift-ingress-canary/ingress-canary-fgzh7"
May 25 11:18:16.074975 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.074957    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f5aae4f-4930-47cc-be9f-2b971dbd12af-cert\") pod \"ingress-canary-fgzh7\" (UID: \"2f5aae4f-4930-47cc-be9f-2b971dbd12af\") " pod="openshift-ingress-canary/ingress-canary-fgzh7"
May 25 11:18:16.081240 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.081220    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwm24\" (UniqueName: \"kubernetes.io/projected/2f5aae4f-4930-47cc-be9f-2b971dbd12af-kube-api-access-gwm24\") pod \"ingress-canary-fgzh7\" (UID: \"2f5aae4f-4930-47cc-be9f-2b971dbd12af\") " pod="openshift-ingress-canary/ingress-canary-fgzh7"
May 25 11:18:16.155751 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.155720    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fgzh7"
May 25 11:18:16.202615 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.202585    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ggmkh"]
May 25 11:18:16.204618 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.204595    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pw56p"]
May 25 11:18:16.208079 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:16.207967    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32313dcb_927f_4c70_917c_f876bfc59823.slice/crio-97b8d9d140ba6b71d503f1121f69568dbc495a8b769afbce3dabff6795cbae78 WatchSource:0}: Error finding container 97b8d9d140ba6b71d503f1121f69568dbc495a8b769afbce3dabff6795cbae78: Status 404 returned error can't find the container with id 97b8d9d140ba6b71d503f1121f69568dbc495a8b769afbce3dabff6795cbae78
May 25 11:18:16.212681 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:16.212656    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6f9fdb_3081_4d28_b6e5_d2d706f7aeb4.slice/crio-39033f283be043ee214d143736dc2ceec2c541c9e5a97f344080af5e89ac7263 WatchSource:0}: Error finding container 39033f283be043ee214d143736dc2ceec2c541c9e5a97f344080af5e89ac7263: Status 404 returned error can't find the container with id 39033f283be043ee214d143736dc2ceec2c541c9e5a97f344080af5e89ac7263
May 25 11:18:16.283541 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.283512    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fgzh7"]
May 25 11:18:16.287662 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:16.287641    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5aae4f_4930_47cc_be9f_2b971dbd12af.slice/crio-19ae621f2ba45b3e4f8a19d002425210b72508ba63a3997e8511068cded67a1b WatchSource:0}: Error finding container 19ae621f2ba45b3e4f8a19d002425210b72508ba63a3997e8511068cded67a1b: Status 404 returned error can't find the container with id 19ae621f2ba45b3e4f8a19d002425210b72508ba63a3997e8511068cded67a1b
May 25 11:18:16.976173 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.976146    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:16.978656 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.978628    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:18:16.979537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.979518    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:18:16.979628 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:16.979546    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6q22d\""
May 25 11:18:17.157530 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.157420    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fgzh7" event={"ID":"2f5aae4f-4930-47cc-be9f-2b971dbd12af","Type":"ContainerStarted","Data":"19ae621f2ba45b3e4f8a19d002425210b72508ba63a3997e8511068cded67a1b"}
May 25 11:18:17.159291 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.159243    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggmkh" event={"ID":"32313dcb-927f-4c70-917c-f876bfc59823","Type":"ContainerStarted","Data":"833d603a424157e5c08c0e7eda6163bf6d0bda7b40c354c0326540865c89fbfe"}
May 25 11:18:17.159421 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.159297    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggmkh" event={"ID":"32313dcb-927f-4c70-917c-f876bfc59823","Type":"ContainerStarted","Data":"7bc7e92bd0faf198364f777828a0f0f7dbe45801ce09fcdc800b4225528f974b"}
May 25 11:18:17.159421 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.159346    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggmkh" event={"ID":"32313dcb-927f-4c70-917c-f876bfc59823","Type":"ContainerStarted","Data":"97b8d9d140ba6b71d503f1121f69568dbc495a8b769afbce3dabff6795cbae78"}
May 25 11:18:17.160542 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.160514    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pw56p" event={"ID":"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4","Type":"ContainerStarted","Data":"39033f283be043ee214d143736dc2ceec2c541c9e5a97f344080af5e89ac7263"}
May 25 11:18:17.307948 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.307745    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"]
May 25 11:18:17.332192 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.332161    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"]
May 25 11:18:17.332333 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.332237    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"
May 25 11:18:17.334629 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.334581    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
May 25 11:18:17.335055 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.334876    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-24bnm\""
May 25 11:18:17.482446 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.482403    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b3de746-239e-44c7-94bd-60d0a466b736-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5\" (UID: \"3b3de746-239e-44c7-94bd-60d0a466b736\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"
May 25 11:18:17.582921 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.582888    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b3de746-239e-44c7-94bd-60d0a466b736-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5\" (UID: \"3b3de746-239e-44c7-94bd-60d0a466b736\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"
May 25 11:18:17.586655 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.586636    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/3b3de746-239e-44c7-94bd-60d0a466b736-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5\" (UID: \"3b3de746-239e-44c7-94bd-60d0a466b736\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"
May 25 11:18:17.643746 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:17.643713    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"
May 25 11:18:18.640831 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:18.640806    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"]
May 25 11:18:18.647320 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:18.647295    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b3de746_239e_44c7_94bd_60d0a466b736.slice/crio-82697a5007714e30e548b1881ff519e47d69f80413e7e42e394d361f3d3ada66 WatchSource:0}: Error finding container 82697a5007714e30e548b1881ff519e47d69f80413e7e42e394d361f3d3ada66: Status 404 returned error can't find the container with id 82697a5007714e30e548b1881ff519e47d69f80413e7e42e394d361f3d3ada66
May 25 11:18:19.166931 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:19.166623    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pw56p" event={"ID":"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4","Type":"ContainerStarted","Data":"aae685b22fa0d063b7cfeca8e848f1288b6c5008f4c1bd7f38227f11d12d36db"}
May 25 11:18:19.166931 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:19.166882    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pw56p"
May 25 11:18:19.166931 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:19.166903    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pw56p" event={"ID":"7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4","Type":"ContainerStarted","Data":"27d698351206ff22f6dd2c9a9d9a0631b83a7041ed9536605c43f24f0422f0ca"}
May 25 11:18:19.168155 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:19.168119    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fgzh7" event={"ID":"2f5aae4f-4930-47cc-be9f-2b971dbd12af","Type":"ContainerStarted","Data":"7e2bba7eb7a5945df127246d0591b6d04aaa4dfed1ac7d80c0932e4d7cd047a2"}
May 25 11:18:19.169165 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:19.169146    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5" event={"ID":"3b3de746-239e-44c7-94bd-60d0a466b736","Type":"ContainerStarted","Data":"82697a5007714e30e548b1881ff519e47d69f80413e7e42e394d361f3d3ada66"}
May 25 11:18:19.195843 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:19.195784    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-pw56p" podStartSLOduration=1.932418159 podStartE2EDuration="4.195771707s" podCreationTimestamp="2026-05-25 11:18:15 +0000 UTC" firstStartedPulling="2026-05-25 11:18:16.21464686 +0000 UTC m=+44.844807648" lastFinishedPulling="2026-05-25 11:18:18.478000395 +0000 UTC m=+47.108161196" observedRunningTime="2026-05-25 11:18:19.19446605 +0000 UTC m=+47.824626860" watchObservedRunningTime="2026-05-25 11:18:19.195771707 +0000 UTC m=+47.825932553"
May 25 11:18:19.223202 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:19.223163    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fgzh7" podStartSLOduration=2.028025983 podStartE2EDuration="4.223148281s" podCreationTimestamp="2026-05-25 11:18:15 +0000 UTC" firstStartedPulling="2026-05-25 11:18:16.289453452 +0000 UTC m=+44.919614240" lastFinishedPulling="2026-05-25 11:18:18.484575747 +0000 UTC m=+47.114736538" observedRunningTime="2026-05-25 11:18:19.222533875 +0000 UTC m=+47.852694686" watchObservedRunningTime="2026-05-25 11:18:19.223148281 +0000 UTC m=+47.853309095"
May 25 11:18:20.174719 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:20.174678    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ggmkh" event={"ID":"32313dcb-927f-4c70-917c-f876bfc59823","Type":"ContainerStarted","Data":"cf2362813938cedd06b41160b0f1faec1b59b3deb3a01abab8d4703ad7239dcf"}
May 25 11:18:20.196594 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:20.196550    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ggmkh" podStartSLOduration=1.804744729 podStartE2EDuration="5.19653447s" podCreationTimestamp="2026-05-25 11:18:15 +0000 UTC" firstStartedPulling="2026-05-25 11:18:16.304774667 +0000 UTC m=+44.934935454" lastFinishedPulling="2026-05-25 11:18:19.696564407 +0000 UTC m=+48.326725195" observedRunningTime="2026-05-25 11:18:20.195247577 +0000 UTC m=+48.825408388" watchObservedRunningTime="2026-05-25 11:18:20.19653447 +0000 UTC m=+48.826695281"
May 25 11:18:21.178244 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.178205    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5" event={"ID":"3b3de746-239e-44c7-94bd-60d0a466b736","Type":"ContainerStarted","Data":"d75ee92f11e8ebdf35a6fe12ef1a5c5784dc9ffdf2cc98cdebaf758e75324ee4"}
May 25 11:18:21.178617 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.178436    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"
May 25 11:18:21.183549 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.183528    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5"
May 25 11:18:21.201762 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.201723    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5" podStartSLOduration=2.095284686 podStartE2EDuration="4.201710425s" podCreationTimestamp="2026-05-25 11:18:17 +0000 UTC" firstStartedPulling="2026-05-25 11:18:18.64923736 +0000 UTC m=+47.279398148" lastFinishedPulling="2026-05-25 11:18:20.755663096 +0000 UTC m=+49.385823887" observedRunningTime="2026-05-25 11:18:21.201622542 +0000 UTC m=+49.831783353" watchObservedRunningTime="2026-05-25 11:18:21.201710425 +0000 UTC m=+49.831871234"
May 25 11:18:21.420744 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.420683    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"]
May 25 11:18:21.440292 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.440269    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.450438 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.450420    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-l2v5r\""
May 25 11:18:21.450543 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.450464    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
May 25 11:18:21.450629 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.450617    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
May 25 11:18:21.450922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.450909    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
May 25 11:18:21.451447 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.451434    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
May 25 11:18:21.451998 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.451986    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
May 25 11:18:21.455315 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.455297    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"]
May 25 11:18:21.609534 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.609510    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9bd5baa-a1c8-446d-b688-e707d041b40c-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.609636 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.609595    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9bd5baa-a1c8-446d-b688-e707d041b40c-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.609697 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.609632    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9bd5baa-a1c8-446d-b688-e707d041b40c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.609697 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.609658    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26ll5\" (UniqueName: \"kubernetes.io/projected/b9bd5baa-a1c8-446d-b688-e707d041b40c-kube-api-access-26ll5\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.709871 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.709847    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9bd5baa-a1c8-446d-b688-e707d041b40c-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.709968 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.709938    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9bd5baa-a1c8-446d-b688-e707d041b40c-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.710024 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.709973    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9bd5baa-a1c8-446d-b688-e707d041b40c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.710121 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.710101    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-26ll5\" (UniqueName: \"kubernetes.io/projected/b9bd5baa-a1c8-446d-b688-e707d041b40c-kube-api-access-26ll5\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.710717 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.710700    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9bd5baa-a1c8-446d-b688-e707d041b40c-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.713250 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.713225    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9bd5baa-a1c8-446d-b688-e707d041b40c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.713333 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.713286    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b9bd5baa-a1c8-446d-b688-e707d041b40c-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.719793 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.719775    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26ll5\" (UniqueName: \"kubernetes.io/projected/b9bd5baa-a1c8-446d-b688-e707d041b40c-kube-api-access-26ll5\") pod \"prometheus-operator-7779bbdd78-57wdg\" (UID: \"b9bd5baa-a1c8-446d-b688-e707d041b40c\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.748854 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.748826    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"
May 25 11:18:21.871214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:21.871071    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-57wdg"]
May 25 11:18:21.874072 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:21.874043    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9bd5baa_a1c8_446d_b688_e707d041b40c.slice/crio-fad7b8b18a530726439842a43c0e2d6a06381f64c738e8989ba2980f844ff929 WatchSource:0}: Error finding container fad7b8b18a530726439842a43c0e2d6a06381f64c738e8989ba2980f844ff929: Status 404 returned error can't find the container with id fad7b8b18a530726439842a43c0e2d6a06381f64c738e8989ba2980f844ff929
May 25 11:18:22.181296 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:22.181264    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg" event={"ID":"b9bd5baa-a1c8-446d-b688-e707d041b40c","Type":"ContainerStarted","Data":"fad7b8b18a530726439842a43c0e2d6a06381f64c738e8989ba2980f844ff929"}
May 25 11:18:24.186991 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:24.186953    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg" event={"ID":"b9bd5baa-a1c8-446d-b688-e707d041b40c","Type":"ContainerStarted","Data":"ad211bfb4307b0b04eeb9961bda523a88cb3ecfcf1d9417a9084daf28e7937dc"}
May 25 11:18:24.187666 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:24.187000    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg" event={"ID":"b9bd5baa-a1c8-446d-b688-e707d041b40c","Type":"ContainerStarted","Data":"410061a036ae3923422b8ef07f3fea5abfea38bb7896ea3e4b49056e4703df36"}
May 25 11:18:24.214657 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:24.214611    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-57wdg" podStartSLOduration=1.476256112 podStartE2EDuration="3.214596076s" podCreationTimestamp="2026-05-25 11:18:21 +0000 UTC" firstStartedPulling="2026-05-25 11:18:21.875858116 +0000 UTC m=+50.506018904" lastFinishedPulling="2026-05-25 11:18:23.61419808 +0000 UTC m=+52.244358868" observedRunningTime="2026-05-25 11:18:24.213234058 +0000 UTC m=+52.843394869" watchObservedRunningTime="2026-05-25 11:18:24.214596076 +0000 UTC m=+52.844756886"
May 25 11:18:25.846258 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.846225    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"]
May 25 11:18:25.873231 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.873205    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:25.873905 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.873881    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"]
May 25 11:18:25.875971 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.875951    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
May 25 11:18:25.876557 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.876533    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
May 25 11:18:25.876653 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.876537    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-csgjc\""
May 25 11:18:25.897231 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.897037    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"]
May 25 11:18:25.897231 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.897067    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zk687"]
May 25 11:18:25.897231 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.897226    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:25.899864 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.899845    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
May 25 11:18:25.900019 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.900004    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
May 25 11:18:25.900080 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.900030    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
May 25 11:18:25.900137 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.900039    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pldst\""
May 25 11:18:25.921120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.921102    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"]
May 25 11:18:25.921212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.921206    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:25.924201 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.924175    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
May 25 11:18:25.924201 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.924182    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
May 25 11:18:25.924437 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.924421    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lsnt9\""
May 25 11:18:25.924728 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.924716    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
May 25 11:18:25.936704 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936681    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:25.936783 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936708    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-textfile\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:25.936783 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936732    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5mcx\" (UniqueName: \"kubernetes.io/projected/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-kube-api-access-x5mcx\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:25.936783 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936764    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:25.936881 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936836    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-metrics-client-ca\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:25.936881 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936862    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:25.936942 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936880    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hnf\" (UniqueName: \"kubernetes.io/projected/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-api-access-t9hnf\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:25.936976 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936944    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c1a143-cca8-4994-8e03-3d76eb3bb115-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:25.936976 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936967    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90c1a143-cca8-4994-8e03-3d76eb3bb115-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:25.937039 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.936984    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c1a143-cca8-4994-8e03-3d76eb3bb115-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:25.937039 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.937000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-tls\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:25.937098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.937048    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:25.937098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.937081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:25.937164 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.937122    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-accelerators-collector-config\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:25.937164 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.937141    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-root\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:25.937223 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.937164    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:25.937278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.937221    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwcg\" (UniqueName: \"kubernetes.io/projected/90c1a143-cca8-4994-8e03-3d76eb3bb115-kube-api-access-tbwcg\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:25.937278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.937241    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-wtmp\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:25.937278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:25.937275    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-sys\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.037556 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037533    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x5mcx\" (UniqueName: \"kubernetes.io/projected/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-kube-api-access-x5mcx\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.037647 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037565    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.037647 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037599    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-metrics-client-ca\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.037647 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037620    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.037808 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037714    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hnf\" (UniqueName: \"kubernetes.io/projected/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-api-access-t9hnf\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.037808 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037772    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c1a143-cca8-4994-8e03-3d76eb3bb115-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.037808 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037801    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90c1a143-cca8-4994-8e03-3d76eb3bb115-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.037956 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037828    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c1a143-cca8-4994-8e03-3d76eb3bb115-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.037956 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037854    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-tls\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.037956 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037885    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.037956 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037910    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.037956 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037938    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-accelerators-collector-config\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.038214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037961    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-root\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.038214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.037989    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.038214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.038020    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwcg\" (UniqueName: \"kubernetes.io/projected/90c1a143-cca8-4994-8e03-3d76eb3bb115-kube-api-access-tbwcg\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.038214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.038045    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-wtmp\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.038214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.038068    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-sys\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.038214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.038097    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.038475 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.038366    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-metrics-client-ca\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.038475 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:26.038495    2556 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
May 25 11:18:26.038592 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.038507    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90c1a143-cca8-4994-8e03-3d76eb3bb115-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.038592 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:26.038554    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90c1a143-cca8-4994-8e03-3d76eb3bb115-openshift-state-metrics-tls podName:90c1a143-cca8-4994-8e03-3d76eb3bb115 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:26.5385387 +0000 UTC m=+55.168699502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/90c1a143-cca8-4994-8e03-3d76eb3bb115-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-gnt7w" (UID: "90c1a143-cca8-4994-8e03-3d76eb3bb115") : secret "openshift-state-metrics-tls" not found
May 25 11:18:26.038867 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.038803    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.038867 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.038816    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-wtmp\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.039006 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.038984    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.039078 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.039061    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-root\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.039131 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.039110    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-sys\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.039131 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.039111    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.039225 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.039157    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-textfile\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.039365 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.039338    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-accelerators-collector-config\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.039466 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.039413    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-textfile\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.041227 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.041204    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-tls\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.041366 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.041309    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.041439 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.041370    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.041517 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.041456    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90c1a143-cca8-4994-8e03-3d76eb3bb115-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.041575 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.041523    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.054726 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.054706    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5mcx\" (UniqueName: \"kubernetes.io/projected/5d4d1611-41f7-4e75-a5a0-de5d58e26b83-kube-api-access-x5mcx\") pod \"node-exporter-zk687\" (UID: \"5d4d1611-41f7-4e75-a5a0-de5d58e26b83\") " pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.054808 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.054727    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hnf\" (UniqueName: \"kubernetes.io/projected/8ab2afe6-3d4c-4562-892d-e5bfa9a67071-kube-api-access-t9hnf\") pod \"kube-state-metrics-54fd6bc5c4-klctg\" (UID: \"8ab2afe6-3d4c-4562-892d-e5bfa9a67071\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.057328 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.057308    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwcg\" (UniqueName: \"kubernetes.io/projected/90c1a143-cca8-4994-8e03-3d76eb3bb115-kube-api-access-tbwcg\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.206587 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.206559    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"
May 25 11:18:26.229341 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.229190    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zk687"
May 25 11:18:26.236106 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:26.236072    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4d1611_41f7_4e75_a5a0_de5d58e26b83.slice/crio-dd71b2d254091214417a1ef41a96cc6748bd90adcc57149ad47acc6b129ec478 WatchSource:0}: Error finding container dd71b2d254091214417a1ef41a96cc6748bd90adcc57149ad47acc6b129ec478: Status 404 returned error can't find the container with id dd71b2d254091214417a1ef41a96cc6748bd90adcc57149ad47acc6b129ec478
May 25 11:18:26.383809 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.383761    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg"]
May 25 11:18:26.406011 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:26.405977    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab2afe6_3d4c_4562_892d_e5bfa9a67071.slice/crio-4b20d7473ef50e9e33015bee33cab2d023bbddb3a2925028a85d5abdee372ef2 WatchSource:0}: Error finding container 4b20d7473ef50e9e33015bee33cab2d023bbddb3a2925028a85d5abdee372ef2: Status 404 returned error can't find the container with id 4b20d7473ef50e9e33015bee33cab2d023bbddb3a2925028a85d5abdee372ef2
May 25 11:18:26.542336 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.542258    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c1a143-cca8-4994-8e03-3d76eb3bb115-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.545391 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.545355    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/90c1a143-cca8-4994-8e03-3d76eb3bb115-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-gnt7w\" (UID: \"90c1a143-cca8-4994-8e03-3d76eb3bb115\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.783595 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.783568    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"
May 25 11:18:26.941914 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:26.941833    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w"]
May 25 11:18:26.945548 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:26.945520    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c1a143_cca8_4994_8e03_3d76eb3bb115.slice/crio-9799a67b3ec2600c8bd4212de430fa0da23e13328580a6e895ab846faa7fe5bf WatchSource:0}: Error finding container 9799a67b3ec2600c8bd4212de430fa0da23e13328580a6e895ab846faa7fe5bf: Status 404 returned error can't find the container with id 9799a67b3ec2600c8bd4212de430fa0da23e13328580a6e895ab846faa7fe5bf
May 25 11:18:27.095557 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.095532    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:18:27.114756 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.112997    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.116626 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.116598    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
May 25 11:18:27.123984 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.123961    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
May 25 11:18:27.124271 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.124168    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
May 25 11:18:27.124923 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.124666    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
May 25 11:18:27.125935 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.125695    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
May 25 11:18:27.125935 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.125778    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
May 25 11:18:27.126083 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.126072    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
May 25 11:18:27.128063 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.126175    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
May 25 11:18:27.128063 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.126625    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-42wgr\""
May 25 11:18:27.128063 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.127920    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
May 25 11:18:27.132815 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.132771    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:18:27.146778 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.146701    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-web-config\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.146778 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.146739    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.146944 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.146849    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.146944 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.146902    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.146944 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.146937    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.147077 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.146970    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrwlb\" (UniqueName: \"kubernetes.io/projected/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-kube-api-access-xrwlb\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.147077 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.147004    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.147182 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.147081    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.147182 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.147120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.147182 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.147165    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.147330 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.147190    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.147330 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.147215    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.147330 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.147239    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-config-out\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.197101 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.197066    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg" event={"ID":"8ab2afe6-3d4c-4562-892d-e5bfa9a67071","Type":"ContainerStarted","Data":"4b20d7473ef50e9e33015bee33cab2d023bbddb3a2925028a85d5abdee372ef2"}
May 25 11:18:27.199123 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.199092    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w" event={"ID":"90c1a143-cca8-4994-8e03-3d76eb3bb115","Type":"ContainerStarted","Data":"b0b6071514eb0cd25bfcc5ce495633de5f3f2d458fc0699ec2bb667ec890bef6"}
May 25 11:18:27.199328 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.199134    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w" event={"ID":"90c1a143-cca8-4994-8e03-3d76eb3bb115","Type":"ContainerStarted","Data":"9799a67b3ec2600c8bd4212de430fa0da23e13328580a6e895ab846faa7fe5bf"}
May 25 11:18:27.200974 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.200935    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zk687" event={"ID":"5d4d1611-41f7-4e75-a5a0-de5d58e26b83","Type":"ContainerStarted","Data":"dd71b2d254091214417a1ef41a96cc6748bd90adcc57149ad47acc6b129ec478"}
May 25 11:18:27.247720 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.247685    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-web-config\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.247835 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.247730    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.247835 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.247780    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.247835 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.247820    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.247988 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.247849    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.247988 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.247882    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xrwlb\" (UniqueName: \"kubernetes.io/projected/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-kube-api-access-xrwlb\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.247988 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.247915    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.247988 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.247959    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.247988 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.247987    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.248204 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.248015    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.248204 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.248041    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.248204 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.248064    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.248204 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.248088    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-config-out\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.249828 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:27.248677    2556 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
May 25 11:18:27.249828 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.248699    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.249828 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:27.248746    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-main-tls podName:f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea nodeName:}" failed. No retries permitted until 2026-05-25 11:18:27.748730663 +0000 UTC m=+56.378891456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea") : secret "alertmanager-main-tls" not found
May 25 11:18:27.249828 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.249289    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.250825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.250757    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.255216 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.255191    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.255323 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.255222    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.259360 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.257649    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.259360 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.257715    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrwlb\" (UniqueName: \"kubernetes.io/projected/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-kube-api-access-xrwlb\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.259360 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.258175    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-config-volume\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.259360 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.258218    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.259360 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.258474    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-config-out\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.260218 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.260167    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-web-config\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.260402 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.260381    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.751171 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.751086    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:27.754590 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:27.754559    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea\") " pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:28.029086 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:28.028914    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
May 25 11:18:28.208993 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:28.208965    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w" event={"ID":"90c1a143-cca8-4994-8e03-3d76eb3bb115","Type":"ContainerStarted","Data":"c64aa5cc6301a8e0bdb6c774cb24815bec321f168bc39429ac1008f3ca6c732b"}
May 25 11:18:28.210576 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:28.210537    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zk687" event={"ID":"5d4d1611-41f7-4e75-a5a0-de5d58e26b83","Type":"ContainerStarted","Data":"8536046784c472bca90382336259f724dbb9c8ff6f9ffa05156607ee9fb61968"}
May 25 11:18:28.212112 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:28.212086    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg" event={"ID":"8ab2afe6-3d4c-4562-892d-e5bfa9a67071","Type":"ContainerStarted","Data":"fcb98ea99493324ea59f77da0930c1ed0ef929ef163f0c54d4a67a89036620dd"}
May 25 11:18:28.233180 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:28.233159    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
May 25 11:18:28.234833 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:28.234809    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bf0a0c_b3df_4d4f_aefb_20728d6f37ea.slice/crio-fb28e5361e37204689ff1161a6f36edc989afe4e8a5d3d796ba4b088403dd48d WatchSource:0}: Error finding container fb28e5361e37204689ff1161a6f36edc989afe4e8a5d3d796ba4b088403dd48d: Status 404 returned error can't find the container with id fb28e5361e37204689ff1161a6f36edc989afe4e8a5d3d796ba4b088403dd48d
May 25 11:18:29.177801 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.177753    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pw56p"
May 25 11:18:29.217512 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.217461    2556 generic.go:358] "Generic (PLEG): container finished" podID="5d4d1611-41f7-4e75-a5a0-de5d58e26b83" containerID="8536046784c472bca90382336259f724dbb9c8ff6f9ffa05156607ee9fb61968" exitCode=0
May 25 11:18:29.217686 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.217575    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zk687" event={"ID":"5d4d1611-41f7-4e75-a5a0-de5d58e26b83","Type":"ContainerDied","Data":"8536046784c472bca90382336259f724dbb9c8ff6f9ffa05156607ee9fb61968"}
May 25 11:18:29.219953 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.219926    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg" event={"ID":"8ab2afe6-3d4c-4562-892d-e5bfa9a67071","Type":"ContainerStarted","Data":"0525c82e59bfc245d84a64acfbaf88dd2d77defbec59e85b953b56a04055a507"}
May 25 11:18:29.220076 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.219960    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg" event={"ID":"8ab2afe6-3d4c-4562-892d-e5bfa9a67071","Type":"ContainerStarted","Data":"5de7f4d9b550cc9cf9d0d699902f9d9985db04acd247e086cbbff09ddc971ae5"}
May 25 11:18:29.221390 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.221339    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea","Type":"ContainerStarted","Data":"fb28e5361e37204689ff1161a6f36edc989afe4e8a5d3d796ba4b088403dd48d"}
May 25 11:18:29.260183 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.260142    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-klctg" podStartSLOduration=2.583098075 podStartE2EDuration="4.260129466s" podCreationTimestamp="2026-05-25 11:18:25 +0000 UTC" firstStartedPulling="2026-05-25 11:18:26.407823866 +0000 UTC m=+55.037984655" lastFinishedPulling="2026-05-25 11:18:28.084855253 +0000 UTC m=+56.715016046" observedRunningTime="2026-05-25 11:18:29.25941266 +0000 UTC m=+57.889573473" watchObservedRunningTime="2026-05-25 11:18:29.260129466 +0000 UTC m=+57.890290276"
May 25 11:18:29.933602 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.932185    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"]
May 25 11:18:29.935222 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.935202    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:29.938297 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.938276    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
May 25 11:18:29.938397 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.938323    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2p48dd26du4a3\""
May 25 11:18:29.938546 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.938452    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
May 25 11:18:29.939320 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.939299    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-r8wcz\""
May 25 11:18:29.939320 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.939317    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
May 25 11:18:29.939624 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.939317    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
May 25 11:18:29.939624 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.939299    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
May 25 11:18:29.948654 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.948631    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"]
May 25 11:18:29.968238 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.968215    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:29.968326 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.968247    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:29.968326 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.968277    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-tls\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:29.968394 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.968350    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:29.968424 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.968401    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9a65038-2eab-4247-9284-b3ff2a36b89c-metrics-client-ca\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:29.968457 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.968423    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hk76\" (UniqueName: \"kubernetes.io/projected/c9a65038-2eab-4247-9284-b3ff2a36b89c-kube-api-access-6hk76\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:29.968518 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.968464    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:29.968561 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:29.968517    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-grpc-tls\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.069331 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.069298    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.069428 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.069338    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.069428 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.069361    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-tls\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.069428 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.069383    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.069428 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.069420    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9a65038-2eab-4247-9284-b3ff2a36b89c-metrics-client-ca\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.069657 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.069443    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6hk76\" (UniqueName: \"kubernetes.io/projected/c9a65038-2eab-4247-9284-b3ff2a36b89c-kube-api-access-6hk76\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.069657 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.069473    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.069657 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.069526    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-grpc-tls\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.070220 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.070172    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c9a65038-2eab-4247-9284-b3ff2a36b89c-metrics-client-ca\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.071700 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.071672    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.071971 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.071950    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-grpc-tls\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.072135 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.072119    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-tls\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.072219 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.072202    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.072309 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.072290    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.072647 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.072630    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c9a65038-2eab-4247-9284-b3ff2a36b89c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.077604 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.077585    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hk76\" (UniqueName: \"kubernetes.io/projected/c9a65038-2eab-4247-9284-b3ff2a36b89c-kube-api-access-6hk76\") pod \"thanos-querier-6b7cd5f99d-68bll\" (UID: \"c9a65038-2eab-4247-9284-b3ff2a36b89c\") " pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.229615 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.229574    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w" event={"ID":"90c1a143-cca8-4994-8e03-3d76eb3bb115","Type":"ContainerStarted","Data":"4aeb90e948b4b05a14aa2b28d1d3dfc8da637b5e102789f487479c602d756ee9"}
May 25 11:18:30.231672 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.231641    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zk687" event={"ID":"5d4d1611-41f7-4e75-a5a0-de5d58e26b83","Type":"ContainerStarted","Data":"f1b889951cc5fa5339e25dbc237dec46dcec8d17386f995e0ecddac5bacc0d32"}
May 25 11:18:30.231672 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.231671    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zk687" event={"ID":"5d4d1611-41f7-4e75-a5a0-de5d58e26b83","Type":"ContainerStarted","Data":"085280a21685cda9e8ff6f2ef6447553990f5c292e4b8314a62c104e8ac7ddc7"}
May 25 11:18:30.231885 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.231864    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6cd8f9d848-f46g9"]
May 25 11:18:30.233009 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.232989    2556 generic.go:358] "Generic (PLEG): container finished" podID="f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea" containerID="3a2645bc6b4f0094683b3f78cee93d5e24d5e33dfac740187bbd6e101a14190c" exitCode=0
May 25 11:18:30.233894 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.233872    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea","Type":"ContainerDied","Data":"3a2645bc6b4f0094683b3f78cee93d5e24d5e33dfac740187bbd6e101a14190c"}
May 25 11:18:30.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.234196    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.237303 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.237193    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
May 25 11:18:30.237303 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.237237    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
May 25 11:18:30.237523 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.237336    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
May 25 11:18:30.237523 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.237395    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
May 25 11:18:30.237523 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.237459    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ershmeom2co0g\""
May 25 11:18:30.238204 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.238185    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-bp7gq\""
May 25 11:18:30.245395 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.245375    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cd8f9d848-f46g9"]
May 25 11:18:30.250792 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.250750    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-gnt7w" podStartSLOduration=3.516393931 podStartE2EDuration="5.25073613s" podCreationTimestamp="2026-05-25 11:18:25 +0000 UTC" firstStartedPulling="2026-05-25 11:18:28.182983307 +0000 UTC m=+56.813144099" lastFinishedPulling="2026-05-25 11:18:29.917325505 +0000 UTC m=+58.547486298" observedRunningTime="2026-05-25 11:18:30.249911829 +0000 UTC m=+58.880072638" watchObservedRunningTime="2026-05-25 11:18:30.25073613 +0000 UTC m=+58.880896941"
May 25 11:18:30.271034 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.271011    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg99j\" (UniqueName: \"kubernetes.io/projected/0277fb40-a9db-499f-aeb8-2e3ff321f967-kube-api-access-zg99j\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.271133 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.271056    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0277fb40-a9db-499f-aeb8-2e3ff321f967-secret-metrics-server-tls\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.271133 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.271123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0277fb40-a9db-499f-aeb8-2e3ff321f967-metrics-server-audit-profiles\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.271241 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.271201    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0277fb40-a9db-499f-aeb8-2e3ff321f967-client-ca-bundle\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.271416 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.271385    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0277fb40-a9db-499f-aeb8-2e3ff321f967-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.271526 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.271422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0277fb40-a9db-499f-aeb8-2e3ff321f967-audit-log\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.271764 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.271746    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0277fb40-a9db-499f-aeb8-2e3ff321f967-secret-metrics-server-client-certs\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.276785 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.276752    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zk687" podStartSLOduration=3.436299531 podStartE2EDuration="5.276742953s" podCreationTimestamp="2026-05-25 11:18:25 +0000 UTC" firstStartedPulling="2026-05-25 11:18:26.237637579 +0000 UTC m=+54.867798367" lastFinishedPulling="2026-05-25 11:18:28.078080985 +0000 UTC m=+56.708241789" observedRunningTime="2026-05-25 11:18:30.275857275 +0000 UTC m=+58.906018107" watchObservedRunningTime="2026-05-25 11:18:30.276742953 +0000 UTC m=+58.906903766"
May 25 11:18:30.282955 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.282940    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:30.373123 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.373093    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zg99j\" (UniqueName: \"kubernetes.io/projected/0277fb40-a9db-499f-aeb8-2e3ff321f967-kube-api-access-zg99j\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.373225 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.373129    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0277fb40-a9db-499f-aeb8-2e3ff321f967-secret-metrics-server-tls\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.373284 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.373233    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0277fb40-a9db-499f-aeb8-2e3ff321f967-metrics-server-audit-profiles\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.373284 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.373273    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0277fb40-a9db-499f-aeb8-2e3ff321f967-client-ca-bundle\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.373403 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.373312    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0277fb40-a9db-499f-aeb8-2e3ff321f967-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.373403 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.373337    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0277fb40-a9db-499f-aeb8-2e3ff321f967-audit-log\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.373521 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.373412    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0277fb40-a9db-499f-aeb8-2e3ff321f967-secret-metrics-server-client-certs\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.374251 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.374221    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/0277fb40-a9db-499f-aeb8-2e3ff321f967-audit-log\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.374369 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.374292    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0277fb40-a9db-499f-aeb8-2e3ff321f967-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.374506 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.374468    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/0277fb40-a9db-499f-aeb8-2e3ff321f967-metrics-server-audit-profiles\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.375719 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.375693    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/0277fb40-a9db-499f-aeb8-2e3ff321f967-secret-metrics-server-tls\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.375906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.375892    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/0277fb40-a9db-499f-aeb8-2e3ff321f967-secret-metrics-server-client-certs\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.376334 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.376317    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/0277fb40-a9db-499f-aeb8-2e3ff321f967-client-ca-bundle\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.382064 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.382039    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg99j\" (UniqueName: \"kubernetes.io/projected/0277fb40-a9db-499f-aeb8-2e3ff321f967-kube-api-access-zg99j\") pod \"metrics-server-6cd8f9d848-f46g9\" (UID: \"0277fb40-a9db-499f-aeb8-2e3ff321f967\") " pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.415628 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.415602    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"]
May 25 11:18:30.418232 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:30.418208    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a65038_2eab_4247_9284_b3ff2a36b89c.slice/crio-a26a531c575eca7d1b5983c41e1cd6d8859a17860118e0f2fd2dd358c7a2f8a4 WatchSource:0}: Error finding container a26a531c575eca7d1b5983c41e1cd6d8859a17860118e0f2fd2dd358c7a2f8a4: Status 404 returned error can't find the container with id a26a531c575eca7d1b5983c41e1cd6d8859a17860118e0f2fd2dd358c7a2f8a4
May 25 11:18:30.545727 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.545668    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:30.602330 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.602298    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"]
May 25 11:18:30.605127 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.605107    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"
May 25 11:18:30.609002 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.608971    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
May 25 11:18:30.609096 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.609016    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-9tckh\""
May 25 11:18:30.620049 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.620024    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"]
May 25 11:18:30.673538 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.673513    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6cd8f9d848-f46g9"]
May 25 11:18:30.675186 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.675162    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2772063-b043-4d74-996e-14736d635b27-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-5cd8r\" (UID: \"b2772063-b043-4d74-996e-14736d635b27\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"
May 25 11:18:30.675383 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:30.675364    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0277fb40_a9db_499f_aeb8_2e3ff321f967.slice/crio-177c014e7b4f9b0cdc7f65db1bd5666df43ca158ab76ff6139d6c0fc30e872ec WatchSource:0}: Error finding container 177c014e7b4f9b0cdc7f65db1bd5666df43ca158ab76ff6139d6c0fc30e872ec: Status 404 returned error can't find the container with id 177c014e7b4f9b0cdc7f65db1bd5666df43ca158ab76ff6139d6c0fc30e872ec
May 25 11:18:30.776527 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:30.776470    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2772063-b043-4d74-996e-14736d635b27-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-5cd8r\" (UID: \"b2772063-b043-4d74-996e-14736d635b27\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"
May 25 11:18:30.776683 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:30.776640    2556 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
May 25 11:18:30.776747 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:18:30.776719    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2772063-b043-4d74-996e-14736d635b27-monitoring-plugin-cert podName:b2772063-b043-4d74-996e-14736d635b27 nodeName:}" failed. No retries permitted until 2026-05-25 11:18:31.276699055 +0000 UTC m=+59.906859843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/b2772063-b043-4d74-996e-14736d635b27-monitoring-plugin-cert") pod "monitoring-plugin-665c9c8584-5cd8r" (UID: "b2772063-b043-4d74-996e-14736d635b27") : secret "monitoring-plugin-cert" not found
May 25 11:18:31.133168 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.133139    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-794p7"
May 25 11:18:31.242631 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.242594    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9" event={"ID":"0277fb40-a9db-499f-aeb8-2e3ff321f967","Type":"ContainerStarted","Data":"177c014e7b4f9b0cdc7f65db1bd5666df43ca158ab76ff6139d6c0fc30e872ec"}
May 25 11:18:31.244574 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.244528    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll" event={"ID":"c9a65038-2eab-4247-9284-b3ff2a36b89c","Type":"ContainerStarted","Data":"a26a531c575eca7d1b5983c41e1cd6d8859a17860118e0f2fd2dd358c7a2f8a4"}
May 25 11:18:31.281423 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.281386    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2772063-b043-4d74-996e-14736d635b27-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-5cd8r\" (UID: \"b2772063-b043-4d74-996e-14736d635b27\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"
May 25 11:18:31.291320 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.291295    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b2772063-b043-4d74-996e-14736d635b27-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-5cd8r\" (UID: \"b2772063-b043-4d74-996e-14736d635b27\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"
May 25 11:18:31.515252 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.515055    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"
May 25 11:18:31.881028 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.880964    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-7567745db7-5lrgw"]
May 25 11:18:31.884031 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.884011    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:31.902313 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.902296    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
May 25 11:18:31.903135 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.903120    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7zphw\""
May 25 11:18:31.903356 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.903343    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
May 25 11:18:31.904073 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.904058    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
May 25 11:18:31.904438 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.904399    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
May 25 11:18:31.904548 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.904417    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
May 25 11:18:31.904548 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.904452    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
May 25 11:18:31.904548 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.904438    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
May 25 11:18:31.918390 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.918365    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
May 25 11:18:31.930760 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.930738    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7567745db7-5lrgw"]
May 25 11:18:31.987776 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.987753    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-serving-cert\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:31.987899 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.987787    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-service-ca\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:31.987899 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.987819    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-trusted-ca-bundle\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:31.987899 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.987889    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-oauth-config\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:31.988062 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.987914    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-oauth-serving-cert\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:31.988062 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.988022    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwdf6\" (UniqueName: \"kubernetes.io/projected/5314c586-8568-4b03-998a-b1575f393a30-kube-api-access-cwdf6\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:31.988062 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:31.988056    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-console-config\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.088833 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.088799    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cwdf6\" (UniqueName: \"kubernetes.io/projected/5314c586-8568-4b03-998a-b1575f393a30-kube-api-access-cwdf6\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.088987 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.088847    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-console-config\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.088987 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.088881    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-serving-cert\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.088987 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.088916    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-service-ca\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.088987 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.088955    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-trusted-ca-bundle\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.089209 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.089008    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-oauth-config\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.089446 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.089417    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-oauth-serving-cert\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.089818 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.089792    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-service-ca\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.089973 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.089889    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-console-config\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.090174 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.090147    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-trusted-ca-bundle\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.090246 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.090169    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-oauth-serving-cert\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.092033 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.091991    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-serving-cert\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.093498 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.093465    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-oauth-config\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.101704 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.101681    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwdf6\" (UniqueName: \"kubernetes.io/projected/5314c586-8568-4b03-998a-b1575f393a30-kube-api-access-cwdf6\") pod \"console-7567745db7-5lrgw\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") " pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.193758 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.193729    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:32.508550 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.508450    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:18:32.512724 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.512686    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.517922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.517726    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
May 25 11:18:32.517922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.517755    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
May 25 11:18:32.518092 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.517998    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
May 25 11:18:32.518092 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.518025    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
May 25 11:18:32.518726 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.518236    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
May 25 11:18:32.518726 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.518521    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
May 25 11:18:32.518726 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.518522    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6ae227o5sq1vs\""
May 25 11:18:32.518726 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.518576    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
May 25 11:18:32.519012 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.518922    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
May 25 11:18:32.519012 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.518985    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
May 25 11:18:32.519381 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.519362    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
May 25 11:18:32.525694 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.525545    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hhjgv\""
May 25 11:18:32.525694 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.525637    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
May 25 11:18:32.531757 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.531732    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
May 25 11:18:32.551317 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.551296    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:18:32.593777 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.593741    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q42n9\" (UniqueName: \"kubernetes.io/projected/2270ae61-5ab6-41f2-8203-77e458a8d22a-kube-api-access-q42n9\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.593777 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.593781    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-web-config\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.593998 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.593808    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.593998 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.593834    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.593998 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.593861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2270ae61-5ab6-41f2-8203-77e458a8d22a-config-out\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.593998 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.593898    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-config\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.593998 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.593919    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.593998 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.593943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594004    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594032    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594059    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2270ae61-5ab6-41f2-8203-77e458a8d22a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594093    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594118    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594150    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2270ae61-5ab6-41f2-8203-77e458a8d22a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594457 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594457 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594284    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594457 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594316    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.594457 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.594347    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.695301 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695270    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.695508 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695308    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.695508 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695338    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.695508 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695365    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q42n9\" (UniqueName: \"kubernetes.io/projected/2270ae61-5ab6-41f2-8203-77e458a8d22a-kube-api-access-q42n9\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.695508 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695393    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-web-config\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.695508 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695422    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.695508 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695448    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695855    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2270ae61-5ab6-41f2-8203-77e458a8d22a-config-out\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695907    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-config\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695934    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.695967    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696048    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696079    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696114    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2270ae61-5ab6-41f2-8203-77e458a8d22a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696139    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696169    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696208    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2270ae61-5ab6-41f2-8203-77e458a8d22a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696241    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696256    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696341    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.696607 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696435    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.697339 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.696800    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2270ae61-5ab6-41f2-8203-77e458a8d22a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.697339 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.697162    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.699432 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.699405    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.699728 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.699681    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2270ae61-5ab6-41f2-8203-77e458a8d22a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.700167 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.700143    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.701229 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.701199    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.702270 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.702222    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2270ae61-5ab6-41f2-8203-77e458a8d22a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.702356 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.702282    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-web-config\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.702430 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.702408    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2270ae61-5ab6-41f2-8203-77e458a8d22a-config-out\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.702626 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.702585    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-config\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.702752 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.702732    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.703044 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.703022    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.703204 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.703188    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.703472 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.703451    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2270ae61-5ab6-41f2-8203-77e458a8d22a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.710666 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.710630    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q42n9\" (UniqueName: \"kubernetes.io/projected/2270ae61-5ab6-41f2-8203-77e458a8d22a-kube-api-access-q42n9\") pod \"prometheus-k8s-0\" (UID: \"2270ae61-5ab6-41f2-8203-77e458a8d22a\") " pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:32.827881 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:32.827811    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:33.095209 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.095146    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7567745db7-5lrgw"]
May 25 11:18:33.115182 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.115134    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"]
May 25 11:18:33.126578 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:33.126547    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2772063_b043_4d74_996e_14736d635b27.slice/crio-9d61d869c4272939e07f8e986ef1e578600a91fdd9d7dfae19d6a8ae6a7b57c1 WatchSource:0}: Error finding container 9d61d869c4272939e07f8e986ef1e578600a91fdd9d7dfae19d6a8ae6a7b57c1: Status 404 returned error can't find the container with id 9d61d869c4272939e07f8e986ef1e578600a91fdd9d7dfae19d6a8ae6a7b57c1
May 25 11:18:33.162324 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.162296    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
May 25 11:18:33.167642 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:33.167618    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2270ae61_5ab6_41f2_8203_77e458a8d22a.slice/crio-0597351db5451a95ea163131908b62455be938b2b17a38052686dbad0abc22da WatchSource:0}: Error finding container 0597351db5451a95ea163131908b62455be938b2b17a38052686dbad0abc22da: Status 404 returned error can't find the container with id 0597351db5451a95ea163131908b62455be938b2b17a38052686dbad0abc22da
May 25 11:18:33.254011 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.253979    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9" event={"ID":"0277fb40-a9db-499f-aeb8-2e3ff321f967","Type":"ContainerStarted","Data":"f41f7483507c1c1a3a16e2276aa5f699d72ee22d4a5e499448a5efd1a4a92344"}
May 25 11:18:33.255878 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.255800    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll" event={"ID":"c9a65038-2eab-4247-9284-b3ff2a36b89c","Type":"ContainerStarted","Data":"08a6d405c796914675b736896c1cb1ee0ce75a7661d76c106991076f6f4c6ab7"}
May 25 11:18:33.255878 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.255826    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll" event={"ID":"c9a65038-2eab-4247-9284-b3ff2a36b89c","Type":"ContainerStarted","Data":"a3cf42021eca4530985cf7074c1ee18be7252d462067ed8a3dfe2150cc6d7d0e"}
May 25 11:18:33.255878 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.255835    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll" event={"ID":"c9a65038-2eab-4247-9284-b3ff2a36b89c","Type":"ContainerStarted","Data":"08f41d276151b3fecc6c02738d182b8e715dfe5b09c8146820f38ad36f5964a4"}
May 25 11:18:33.256802 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.256780    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567745db7-5lrgw" event={"ID":"5314c586-8568-4b03-998a-b1575f393a30","Type":"ContainerStarted","Data":"0637ab4ba18dbc452ff7a4eb1322caa457cc5c23974176d3b2ac7b19fe09bb79"}
May 25 11:18:33.257722 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.257703    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r" event={"ID":"b2772063-b043-4d74-996e-14736d635b27","Type":"ContainerStarted","Data":"9d61d869c4272939e07f8e986ef1e578600a91fdd9d7dfae19d6a8ae6a7b57c1"}
May 25 11:18:33.259496 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.259462    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea","Type":"ContainerStarted","Data":"c0104e4c1f705c4ede4e79727a32cf48c11abe301843af262352e718b0ec21a3"}
May 25 11:18:33.259591 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.259513    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea","Type":"ContainerStarted","Data":"696abc0c2ac5c5b2218f98d95a7182664446ca1e95bacdfe897d5508398354da"}
May 25 11:18:33.259591 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.259527    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea","Type":"ContainerStarted","Data":"ae90d4dbcde43e28573e3babd6df5bbd30890b06bab7743073fe675504669aa4"}
May 25 11:18:33.260848 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.260751    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2270ae61-5ab6-41f2-8203-77e458a8d22a","Type":"ContainerStarted","Data":"ccaf701c7c941b5e4bd99e6f80c2de278149b1c10c20bbd71ae5d6f9764ebc3b"}
May 25 11:18:33.260848 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.260775    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2270ae61-5ab6-41f2-8203-77e458a8d22a","Type":"ContainerStarted","Data":"0597351db5451a95ea163131908b62455be938b2b17a38052686dbad0abc22da"}
May 25 11:18:33.466058 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:33.466003    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9" podStartSLOduration=1.26806113 podStartE2EDuration="3.465986723s" podCreationTimestamp="2026-05-25 11:18:30 +0000 UTC" firstStartedPulling="2026-05-25 11:18:30.677261237 +0000 UTC m=+59.307422024" lastFinishedPulling="2026-05-25 11:18:32.875186819 +0000 UTC m=+61.505347617" observedRunningTime="2026-05-25 11:18:33.29064212 +0000 UTC m=+61.920802934" watchObservedRunningTime="2026-05-25 11:18:33.465986723 +0000 UTC m=+62.096147534"
May 25 11:18:34.270719 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.269528    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea","Type":"ContainerStarted","Data":"a0db1d67ae8dda872cf9f29fe3a01abab41b58a50c9e85305cb5ad24902a7209"}
May 25 11:18:34.270719 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.269573    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea","Type":"ContainerStarted","Data":"50d520cd718f21d802684276557b2fb9f2b84ce1ebb73ad261f0b21a56cf4080"}
May 25 11:18:34.270719 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.269587    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea","Type":"ContainerStarted","Data":"8f720257b57301af2f1a1b51facc451bbd0624008f8f7d3c7fc9ed121b8a9b93"}
May 25 11:18:34.272927 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.272771    2556 generic.go:358] "Generic (PLEG): container finished" podID="2270ae61-5ab6-41f2-8203-77e458a8d22a" containerID="ccaf701c7c941b5e4bd99e6f80c2de278149b1c10c20bbd71ae5d6f9764ebc3b" exitCode=0
May 25 11:18:34.272927 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.272880    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2270ae61-5ab6-41f2-8203-77e458a8d22a","Type":"ContainerDied","Data":"ccaf701c7c941b5e4bd99e6f80c2de278149b1c10c20bbd71ae5d6f9764ebc3b"}
May 25 11:18:34.277605 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.277540    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll" event={"ID":"c9a65038-2eab-4247-9284-b3ff2a36b89c","Type":"ContainerStarted","Data":"97bd5bd3a8ee6fc4990960a493f737c6628e246d5ce00b6bf8c39f00f570dc85"}
May 25 11:18:34.277605 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.277566    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll" event={"ID":"c9a65038-2eab-4247-9284-b3ff2a36b89c","Type":"ContainerStarted","Data":"44878ed6d81585bd88ac6e7e6e3cc502b553c8ab41f6c00d5306698fde26d453"}
May 25 11:18:34.277605 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.277579    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll" event={"ID":"c9a65038-2eab-4247-9284-b3ff2a36b89c","Type":"ContainerStarted","Data":"f0cec0c76f8dd05af1a9ebeaff82a9ecc1e555ac045e4eb3610dc4d9550ab566"}
May 25 11:18:34.278001 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.277983    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:34.303956 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:34.303909    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.400112225 podStartE2EDuration="7.303897515s" podCreationTimestamp="2026-05-25 11:18:27 +0000 UTC" firstStartedPulling="2026-05-25 11:18:28.237385138 +0000 UTC m=+56.867545945" lastFinishedPulling="2026-05-25 11:18:34.141170442 +0000 UTC m=+62.771331235" observedRunningTime="2026-05-25 11:18:34.301527177 +0000 UTC m=+62.931687987" watchObservedRunningTime="2026-05-25 11:18:34.303897515 +0000 UTC m=+62.934058325"
May 25 11:18:35.282300 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:35.282238    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r" event={"ID":"b2772063-b043-4d74-996e-14736d635b27","Type":"ContainerStarted","Data":"48854bffc108b3fa90f4531a65210b582af2d10f7016155999533e8ee42b4a30"}
May 25 11:18:35.282870 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:35.282833    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"
May 25 11:18:35.288363 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:35.288329    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r"
May 25 11:18:35.326623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:35.326579    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-5cd8r" podStartSLOduration=3.7679946749999997 podStartE2EDuration="5.326566943s" podCreationTimestamp="2026-05-25 11:18:30 +0000 UTC" firstStartedPulling="2026-05-25 11:18:33.128593518 +0000 UTC m=+61.758754306" lastFinishedPulling="2026-05-25 11:18:34.687165778 +0000 UTC m=+63.317326574" observedRunningTime="2026-05-25 11:18:35.326220173 +0000 UTC m=+63.956380983" watchObservedRunningTime="2026-05-25 11:18:35.326566943 +0000 UTC m=+63.956727753"
May 25 11:18:35.326984 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:35.326952    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll" podStartSLOduration=2.7871329339999997 podStartE2EDuration="6.326943655s" podCreationTimestamp="2026-05-25 11:18:29 +0000 UTC" firstStartedPulling="2026-05-25 11:18:30.420104042 +0000 UTC m=+59.050264830" lastFinishedPulling="2026-05-25 11:18:33.959914758 +0000 UTC m=+62.590075551" observedRunningTime="2026-05-25 11:18:34.333533123 +0000 UTC m=+62.963693934" watchObservedRunningTime="2026-05-25 11:18:35.326943655 +0000 UTC m=+63.957104465"
May 25 11:18:37.647362 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:37.647328    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:37.649946 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:37.649920    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
May 25 11:18:37.660520 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:37.660500    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37262c21-a42a-4ff7-90c7-bbe50a05d68b-metrics-certs\") pod \"network-metrics-daemon-8w9qg\" (UID: \"37262c21-a42a-4ff7-90c7-bbe50a05d68b\") " pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:37.748315 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:37.748284    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:37.750826 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:37.750807    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
May 25 11:18:37.761073 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:37.761055    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
May 25 11:18:37.771421 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:37.771402    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82md6\" (UniqueName: \"kubernetes.io/projected/4ed96c55-4683-4498-94f8-a8865cd52373-kube-api-access-82md6\") pod \"network-check-target-2j6dr\" (UID: \"4ed96c55-4683-4498-94f8-a8865cd52373\") " pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:37.894221 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:37.894164    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l78wx\""
May 25 11:18:37.902037 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:37.901897    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8w9qg"
May 25 11:18:38.004201 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.004171    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6q22d\""
May 25 11:18:38.011361 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.011225    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:38.066888 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.066612    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8w9qg"]
May 25 11:18:38.072238 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:38.072032    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37262c21_a42a_4ff7_90c7_bbe50a05d68b.slice/crio-e58f8622a997e0d28a9b628a61787aee1965c112f3f842968d77326ee1c772e6 WatchSource:0}: Error finding container e58f8622a997e0d28a9b628a61787aee1965c112f3f842968d77326ee1c772e6: Status 404 returned error can't find the container with id e58f8622a997e0d28a9b628a61787aee1965c112f3f842968d77326ee1c772e6
May 25 11:18:38.160163 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.160119    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2j6dr"]
May 25 11:18:38.162748 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:18:38.162722    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ed96c55_4683_4498_94f8_a8865cd52373.slice/crio-5ace0d6cc2e301cf7f50ea9f449bc33ab0bc40cc15875f73afa4b19118bafeb7 WatchSource:0}: Error finding container 5ace0d6cc2e301cf7f50ea9f449bc33ab0bc40cc15875f73afa4b19118bafeb7: Status 404 returned error can't find the container with id 5ace0d6cc2e301cf7f50ea9f449bc33ab0bc40cc15875f73afa4b19118bafeb7
May 25 11:18:38.297034 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.297009    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2270ae61-5ab6-41f2-8203-77e458a8d22a","Type":"ContainerStarted","Data":"51b0eda5c187c58159dda5ab485a07d6c601b2e78a032b3e37c6c42b955b4b18"}
May 25 11:18:38.297157 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.297043    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2270ae61-5ab6-41f2-8203-77e458a8d22a","Type":"ContainerStarted","Data":"e74aa41eec440af4c0d08ecaeaaeb20c1ba8dbd57d57f92c1dc558ce21732426"}
May 25 11:18:38.297157 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.297056    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2270ae61-5ab6-41f2-8203-77e458a8d22a","Type":"ContainerStarted","Data":"0b738d852b3132efcd81cc5d8542332587efe2bab80852c70c6e33dc82c74994"}
May 25 11:18:38.297157 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.297064    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2270ae61-5ab6-41f2-8203-77e458a8d22a","Type":"ContainerStarted","Data":"6d291fa7d419c721c59f3e00818bc96e43e854a05706ae7fcc3039c535754d34"}
May 25 11:18:38.298082 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.298051    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2j6dr" event={"ID":"4ed96c55-4683-4498-94f8-a8865cd52373","Type":"ContainerStarted","Data":"5ace0d6cc2e301cf7f50ea9f449bc33ab0bc40cc15875f73afa4b19118bafeb7"}
May 25 11:18:38.299282 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.299256    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8w9qg" event={"ID":"37262c21-a42a-4ff7-90c7-bbe50a05d68b","Type":"ContainerStarted","Data":"e58f8622a997e0d28a9b628a61787aee1965c112f3f842968d77326ee1c772e6"}
May 25 11:18:38.300467 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.300448    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567745db7-5lrgw" event={"ID":"5314c586-8568-4b03-998a-b1575f393a30","Type":"ContainerStarted","Data":"b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1"}
May 25 11:18:38.330379 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:38.330339    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-7567745db7-5lrgw" podStartSLOduration=2.769947266 podStartE2EDuration="7.330328137s" podCreationTimestamp="2026-05-25 11:18:31 +0000 UTC" firstStartedPulling="2026-05-25 11:18:33.107376852 +0000 UTC m=+61.737537646" lastFinishedPulling="2026-05-25 11:18:37.667757717 +0000 UTC m=+66.297918517" observedRunningTime="2026-05-25 11:18:38.329676772 +0000 UTC m=+66.959837574" watchObservedRunningTime="2026-05-25 11:18:38.330328137 +0000 UTC m=+66.960488948"
May 25 11:18:39.311690 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:39.311609    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2270ae61-5ab6-41f2-8203-77e458a8d22a","Type":"ContainerStarted","Data":"af6daed488b95d084550885c2b0e84ab47ef482a71f64ae40b115410230d1581"}
May 25 11:18:39.311690 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:39.311668    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2270ae61-5ab6-41f2-8203-77e458a8d22a","Type":"ContainerStarted","Data":"9145d32888ac00caa961bb644c4e87408f67ee4ef1abe2f2ba42d03982d80ff2"}
May 25 11:18:39.314591 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:39.314544    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8w9qg" event={"ID":"37262c21-a42a-4ff7-90c7-bbe50a05d68b","Type":"ContainerStarted","Data":"9daf518caa180897dcd47a5472b8627b6b5f0a06f5301bf69784cbed16b175ce"}
May 25 11:18:39.345376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:39.345321    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.933865005 podStartE2EDuration="7.345304695s" podCreationTimestamp="2026-05-25 11:18:32 +0000 UTC" firstStartedPulling="2026-05-25 11:18:33.262359094 +0000 UTC m=+61.892519899" lastFinishedPulling="2026-05-25 11:18:37.673798798 +0000 UTC m=+66.303959589" observedRunningTime="2026-05-25 11:18:39.343593989 +0000 UTC m=+67.973754798" watchObservedRunningTime="2026-05-25 11:18:39.345304695 +0000 UTC m=+67.975465506"
May 25 11:18:40.289165 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:40.289136    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6b7cd5f99d-68bll"
May 25 11:18:40.321571 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:40.321531    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8w9qg" event={"ID":"37262c21-a42a-4ff7-90c7-bbe50a05d68b","Type":"ContainerStarted","Data":"37f2d308e1ecef679ae04eb3718e924b5ec1af496d6685b8261307453809f21f"}
May 25 11:18:40.347886 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:40.347828    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8w9qg" podStartSLOduration=67.314680505 podStartE2EDuration="1m8.347810617s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:18:38.074805857 +0000 UTC m=+66.704966645" lastFinishedPulling="2026-05-25 11:18:39.107935954 +0000 UTC m=+67.738096757" observedRunningTime="2026-05-25 11:18:40.345830694 +0000 UTC m=+68.975991504" watchObservedRunningTime="2026-05-25 11:18:40.347810617 +0000 UTC m=+68.977971428"
May 25 11:18:41.326200 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:41.326161    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2j6dr" event={"ID":"4ed96c55-4683-4498-94f8-a8865cd52373","Type":"ContainerStarted","Data":"72afecf6045c45cfbfdea4946e09a5a64d70be1d5b66ea0960015c02b8caebdb"}
May 25 11:18:41.326596 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:41.326228    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:18:41.348716 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:41.348659    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2j6dr" podStartSLOduration=66.59025158 podStartE2EDuration="1m9.348642426s" podCreationTimestamp="2026-05-25 11:17:32 +0000 UTC" firstStartedPulling="2026-05-25 11:18:38.165260559 +0000 UTC m=+66.795421362" lastFinishedPulling="2026-05-25 11:18:40.923651413 +0000 UTC m=+69.553812208" observedRunningTime="2026-05-25 11:18:41.347254029 +0000 UTC m=+69.977414841" watchObservedRunningTime="2026-05-25 11:18:41.348642426 +0000 UTC m=+69.978803250"
May 25 11:18:42.194679 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:42.194649    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:42.194679 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:42.194686    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:42.199456 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:42.199433    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:42.333150 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:42.333124    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:18:42.828774 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:42.828747    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:18:50.546393 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:50.546362    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:18:50.546393 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:18:50.546399    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:19:10.550763 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:10.550728    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:19:10.554572 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:10.554552    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6cd8f9d848-f46g9"
May 25 11:19:12.332633 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:12.332607    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2j6dr"
May 25 11:19:32.828403 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:32.828367    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:19:32.847549 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:32.847526    2556 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:19:33.499784 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:33.499755    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
May 25 11:19:50.513344 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.513304    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"]
May 25 11:19:50.516281 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.516264    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.522729 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.522413    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
May 25 11:19:50.523088 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.523054    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
May 25 11:19:50.523199 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.523062    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
May 25 11:19:50.523199 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.523062    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
May 25 11:19:50.523443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.523428    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8f69j\""
May 25 11:19:50.523857 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.523838    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
May 25 11:19:50.528989 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.528971    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
May 25 11:19:50.541265 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.541241    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"]
May 25 11:19:50.605038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.605012    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvtb4\" (UniqueName: \"kubernetes.io/projected/85b0e62e-f512-492b-8e96-145bf0452b55-kube-api-access-xvtb4\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.605142 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.605046    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-secret-telemeter-client\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.605142 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.605065    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-telemeter-client-tls\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.605211 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.605140    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b0e62e-f512-492b-8e96-145bf0452b55-metrics-client-ca\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.605258 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.605209    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b0e62e-f512-492b-8e96-145bf0452b55-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.605258 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.605246    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b0e62e-f512-492b-8e96-145bf0452b55-serving-certs-ca-bundle\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.605356 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.605270    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-federate-client-tls\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.605406 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.605368    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.705931 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.705904    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b0e62e-f512-492b-8e96-145bf0452b55-metrics-client-ca\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706029 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.705938    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b0e62e-f512-492b-8e96-145bf0452b55-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706029 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.705960    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b0e62e-f512-492b-8e96-145bf0452b55-serving-certs-ca-bundle\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706178 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.706160    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-federate-client-tls\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706244 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.706226    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706296 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.706269    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xvtb4\" (UniqueName: \"kubernetes.io/projected/85b0e62e-f512-492b-8e96-145bf0452b55-kube-api-access-xvtb4\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706349 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.706318    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-secret-telemeter-client\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706400 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.706348    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-telemeter-client-tls\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706709 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.706656    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b0e62e-f512-492b-8e96-145bf0452b55-serving-certs-ca-bundle\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706816 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.706722    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b0e62e-f512-492b-8e96-145bf0452b55-metrics-client-ca\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.706816 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.706798    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b0e62e-f512-492b-8e96-145bf0452b55-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.708612 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.708587    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.708765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.708748    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-federate-client-tls\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.709034 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.709019    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-secret-telemeter-client\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.709092 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.709063    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85b0e62e-f512-492b-8e96-145bf0452b55-telemeter-client-tls\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.717086 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.717062    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvtb4\" (UniqueName: \"kubernetes.io/projected/85b0e62e-f512-492b-8e96-145bf0452b55-kube-api-access-xvtb4\") pod \"telemeter-client-5ffd9b9b68-wsxpv\" (UID: \"85b0e62e-f512-492b-8e96-145bf0452b55\") " pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.825768 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.825723    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"
May 25 11:19:50.954396 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:50.954364    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv"]
May 25 11:19:50.957527 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:19:50.957496    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b0e62e_f512_492b_8e96_145bf0452b55.slice/crio-f39b85dfbd6a5595a2ea94130b6f027acd720c7741403ee9461d301b3e1fce28 WatchSource:0}: Error finding container f39b85dfbd6a5595a2ea94130b6f027acd720c7741403ee9461d301b3e1fce28: Status 404 returned error can't find the container with id f39b85dfbd6a5595a2ea94130b6f027acd720c7741403ee9461d301b3e1fce28
May 25 11:19:51.533112 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:51.533071    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv" event={"ID":"85b0e62e-f512-492b-8e96-145bf0452b55","Type":"ContainerStarted","Data":"f39b85dfbd6a5595a2ea94130b6f027acd720c7741403ee9461d301b3e1fce28"}
May 25 11:19:53.540780 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:53.540743    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv" event={"ID":"85b0e62e-f512-492b-8e96-145bf0452b55","Type":"ContainerStarted","Data":"5eacf1810601a695a42ac38384a93bb9686f63710fe824b86103a028ee4854ea"}
May 25 11:19:53.540780 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:53.540781    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv" event={"ID":"85b0e62e-f512-492b-8e96-145bf0452b55","Type":"ContainerStarted","Data":"bf0681d4545b958127e681780a4f9ce62a305edfe5a319b1218926ccac58eef1"}
May 25 11:19:53.541166 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:53.540790    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv" event={"ID":"85b0e62e-f512-492b-8e96-145bf0452b55","Type":"ContainerStarted","Data":"ebc01185767384c7603235a09f0af7596ac09ceb43788f3296e9d550eb1fe1dc"}
May 25 11:19:53.565192 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:19:53.565147    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5ffd9b9b68-wsxpv" podStartSLOduration=1.7679339 podStartE2EDuration="3.56513274s" podCreationTimestamp="2026-05-25 11:19:50 +0000 UTC" firstStartedPulling="2026-05-25 11:19:50.959171082 +0000 UTC m=+139.589331870" lastFinishedPulling="2026-05-25 11:19:52.756369915 +0000 UTC m=+141.386530710" observedRunningTime="2026-05-25 11:19:53.563781952 +0000 UTC m=+142.193942762" watchObservedRunningTime="2026-05-25 11:19:53.56513274 +0000 UTC m=+142.195293590"
May 25 11:20:04.616765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:04.616734    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7567745db7-5lrgw"]
May 25 11:20:29.636448 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:29.636400    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-7567745db7-5lrgw" podUID="5314c586-8568-4b03-998a-b1575f393a30" containerName="console" containerID="cri-o://b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1" gracePeriod=15
May 25 11:20:29.881633 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:29.881612    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7567745db7-5lrgw_5314c586-8568-4b03-998a-b1575f393a30/console/0.log"
May 25 11:20:29.881735 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:29.881685    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:20:29.999761 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:29.999737    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-trusted-ca-bundle\") pod \"5314c586-8568-4b03-998a-b1575f393a30\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") "
May 25 11:20:29.999878 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:29.999764    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-console-config\") pod \"5314c586-8568-4b03-998a-b1575f393a30\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") "
May 25 11:20:29.999878 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:29.999799    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-serving-cert\") pod \"5314c586-8568-4b03-998a-b1575f393a30\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") "
May 25 11:20:29.999988 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:29.999970    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-oauth-config\") pod \"5314c586-8568-4b03-998a-b1575f393a30\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") "
May 25 11:20:30.000061 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.000046    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-service-ca\") pod \"5314c586-8568-4b03-998a-b1575f393a30\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") "
May 25 11:20:30.000120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.000081    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-oauth-serving-cert\") pod \"5314c586-8568-4b03-998a-b1575f393a30\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") "
May 25 11:20:30.000120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.000093    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-console-config" (OuterVolumeSpecName: "console-config") pod "5314c586-8568-4b03-998a-b1575f393a30" (UID: "5314c586-8568-4b03-998a-b1575f393a30"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:30.000120 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.000103    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwdf6\" (UniqueName: \"kubernetes.io/projected/5314c586-8568-4b03-998a-b1575f393a30-kube-api-access-cwdf6\") pod \"5314c586-8568-4b03-998a-b1575f393a30\" (UID: \"5314c586-8568-4b03-998a-b1575f393a30\") "
May 25 11:20:30.000274 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.000133    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5314c586-8568-4b03-998a-b1575f393a30" (UID: "5314c586-8568-4b03-998a-b1575f393a30"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:30.000373 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.000356    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-trusted-ca-bundle\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:20:30.000442 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.000377    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-console-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:20:30.000512 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.000454    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-service-ca" (OuterVolumeSpecName: "service-ca") pod "5314c586-8568-4b03-998a-b1575f393a30" (UID: "5314c586-8568-4b03-998a-b1575f393a30"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:30.000512 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.000451    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5314c586-8568-4b03-998a-b1575f393a30" (UID: "5314c586-8568-4b03-998a-b1575f393a30"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:20:30.002112 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.002091    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5314c586-8568-4b03-998a-b1575f393a30" (UID: "5314c586-8568-4b03-998a-b1575f393a30"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:20:30.002198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.002116    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5314c586-8568-4b03-998a-b1575f393a30" (UID: "5314c586-8568-4b03-998a-b1575f393a30"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:20:30.002198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.002164    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5314c586-8568-4b03-998a-b1575f393a30-kube-api-access-cwdf6" (OuterVolumeSpecName: "kube-api-access-cwdf6") pod "5314c586-8568-4b03-998a-b1575f393a30" (UID: "5314c586-8568-4b03-998a-b1575f393a30"). InnerVolumeSpecName "kube-api-access-cwdf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:20:30.101347 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.101325    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-service-ca\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:20:30.101347 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.101347    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5314c586-8568-4b03-998a-b1575f393a30-oauth-serving-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:20:30.101470 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.101361    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwdf6\" (UniqueName: \"kubernetes.io/projected/5314c586-8568-4b03-998a-b1575f393a30-kube-api-access-cwdf6\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:20:30.101470 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.101372    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-serving-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:20:30.101470 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.101380    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5314c586-8568-4b03-998a-b1575f393a30-console-oauth-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:20:30.645252 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.645227    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7567745db7-5lrgw_5314c586-8568-4b03-998a-b1575f393a30/console/0.log"
May 25 11:20:30.645613 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.645264    2556 generic.go:358] "Generic (PLEG): container finished" podID="5314c586-8568-4b03-998a-b1575f393a30" containerID="b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1" exitCode=2
May 25 11:20:30.645613 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.645332    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7567745db7-5lrgw"
May 25 11:20:30.645613 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.645355    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567745db7-5lrgw" event={"ID":"5314c586-8568-4b03-998a-b1575f393a30","Type":"ContainerDied","Data":"b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1"}
May 25 11:20:30.645613 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.645399    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7567745db7-5lrgw" event={"ID":"5314c586-8568-4b03-998a-b1575f393a30","Type":"ContainerDied","Data":"0637ab4ba18dbc452ff7a4eb1322caa457cc5c23974176d3b2ac7b19fe09bb79"}
May 25 11:20:30.645613 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.645419    2556 scope.go:117] "RemoveContainer" containerID="b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1"
May 25 11:20:30.654028 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.654005    2556 scope.go:117] "RemoveContainer" containerID="b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1"
May 25 11:20:30.654293 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:20:30.654273    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1\": container with ID starting with b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1 not found: ID does not exist" containerID="b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1"
May 25 11:20:30.654340 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.654301    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1"} err="failed to get container status \"b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1\": rpc error: code = NotFound desc = could not find container \"b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1\": container with ID starting with b7fdb1b69a7908df0e3d669e0e96a43ffa86b94321c7ea80b83ade3a40e51ac1 not found: ID does not exist"
May 25 11:20:30.666240 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.666217    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7567745db7-5lrgw"]
May 25 11:20:30.669742 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:30.669720    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7567745db7-5lrgw"]
May 25 11:20:31.980322 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:31.980285    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5314c586-8568-4b03-998a-b1575f393a30" path="/var/lib/kubelet/pods/5314c586-8568-4b03-998a-b1575f393a30/volumes"
May 25 11:20:44.862153 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:44.862118    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-m5jhd"]
May 25 11:20:44.862649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:44.862438    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5314c586-8568-4b03-998a-b1575f393a30" containerName="console"
May 25 11:20:44.862649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:44.862449    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5314c586-8568-4b03-998a-b1575f393a30" containerName="console"
May 25 11:20:44.862649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:44.862522    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="5314c586-8568-4b03-998a-b1575f393a30" containerName="console"
May 25 11:20:44.865405 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:44.865389    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:44.867997 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:44.867977    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
May 25 11:20:44.875083 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:44.875062    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m5jhd"]
May 25 11:20:45.009921 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.009895    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/08a5630b-4d2a-493b-84d6-56f03d88d747-kubelet-config\") pod \"global-pull-secret-syncer-m5jhd\" (UID: \"08a5630b-4d2a-493b-84d6-56f03d88d747\") " pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.010064 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.009941    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/08a5630b-4d2a-493b-84d6-56f03d88d747-dbus\") pod \"global-pull-secret-syncer-m5jhd\" (UID: \"08a5630b-4d2a-493b-84d6-56f03d88d747\") " pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.010064 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.009973    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08a5630b-4d2a-493b-84d6-56f03d88d747-original-pull-secret\") pod \"global-pull-secret-syncer-m5jhd\" (UID: \"08a5630b-4d2a-493b-84d6-56f03d88d747\") " pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.110672 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.110644    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/08a5630b-4d2a-493b-84d6-56f03d88d747-kubelet-config\") pod \"global-pull-secret-syncer-m5jhd\" (UID: \"08a5630b-4d2a-493b-84d6-56f03d88d747\") " pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.110794 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.110690    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/08a5630b-4d2a-493b-84d6-56f03d88d747-dbus\") pod \"global-pull-secret-syncer-m5jhd\" (UID: \"08a5630b-4d2a-493b-84d6-56f03d88d747\") " pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.110794 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.110718    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08a5630b-4d2a-493b-84d6-56f03d88d747-original-pull-secret\") pod \"global-pull-secret-syncer-m5jhd\" (UID: \"08a5630b-4d2a-493b-84d6-56f03d88d747\") " pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.110794 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.110781    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/08a5630b-4d2a-493b-84d6-56f03d88d747-kubelet-config\") pod \"global-pull-secret-syncer-m5jhd\" (UID: \"08a5630b-4d2a-493b-84d6-56f03d88d747\") " pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.110906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.110839    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/08a5630b-4d2a-493b-84d6-56f03d88d747-dbus\") pod \"global-pull-secret-syncer-m5jhd\" (UID: \"08a5630b-4d2a-493b-84d6-56f03d88d747\") " pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.112846 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.112807    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/08a5630b-4d2a-493b-84d6-56f03d88d747-original-pull-secret\") pod \"global-pull-secret-syncer-m5jhd\" (UID: \"08a5630b-4d2a-493b-84d6-56f03d88d747\") " pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.174873 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.174852    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m5jhd"
May 25 11:20:45.293914 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.293886    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m5jhd"]
May 25 11:20:45.296246 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:20:45.296222    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08a5630b_4d2a_493b_84d6_56f03d88d747.slice/crio-bb8c6e9599ea1a7b1f18c307f1eb8f5e2ebf55d5f68b1d98c89e29b277571b4b WatchSource:0}: Error finding container bb8c6e9599ea1a7b1f18c307f1eb8f5e2ebf55d5f68b1d98c89e29b277571b4b: Status 404 returned error can't find the container with id bb8c6e9599ea1a7b1f18c307f1eb8f5e2ebf55d5f68b1d98c89e29b277571b4b
May 25 11:20:45.690733 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:45.690689    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m5jhd" event={"ID":"08a5630b-4d2a-493b-84d6-56f03d88d747","Type":"ContainerStarted","Data":"bb8c6e9599ea1a7b1f18c307f1eb8f5e2ebf55d5f68b1d98c89e29b277571b4b"}
May 25 11:20:49.704553 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:49.704521    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m5jhd" event={"ID":"08a5630b-4d2a-493b-84d6-56f03d88d747","Type":"ContainerStarted","Data":"aa5ed776f1cff46ecc0f23efa959a357a9c4ab80f753a44ec8b3b7d820c0661c"}
May 25 11:20:49.722618 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:20:49.722575    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m5jhd" podStartSLOduration=1.7461591950000002 podStartE2EDuration="5.722561921s" podCreationTimestamp="2026-05-25 11:20:44 +0000 UTC" firstStartedPulling="2026-05-25 11:20:45.298136722 +0000 UTC m=+193.928297510" lastFinishedPulling="2026-05-25 11:20:49.274539444 +0000 UTC m=+197.904700236" observedRunningTime="2026-05-25 11:20:49.721425836 +0000 UTC m=+198.351586646" watchObservedRunningTime="2026-05-25 11:20:49.722561921 +0000 UTC m=+198.352722731"
May 25 11:22:31.854371 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:22:31.854340    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:22:31.855896 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:22:31.855873    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:22:31.861186 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:22:31.861166    2556 kubelet.go:1628] "Image garbage collection succeeded"
May 25 11:24:09.289158 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.289127    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-696fc77849-c9c6b"]
May 25 11:24:09.291860 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.291845    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:09.294458 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.294439    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-jhhk4\""
May 25 11:24:09.295345 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.295327    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
May 25 11:24:09.295435 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.295363    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
May 25 11:24:09.296187 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.296171    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
May 25 11:24:09.307366 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.307344    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-c9c6b"]
May 25 11:24:09.400838 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.400812    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd326d61-80e0-44e8-be8b-ff6c77610195-cert\") pod \"odh-model-controller-696fc77849-c9c6b\" (UID: \"fd326d61-80e0-44e8-be8b-ff6c77610195\") " pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:09.400960 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.400850    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bmrj\" (UniqueName: \"kubernetes.io/projected/fd326d61-80e0-44e8-be8b-ff6c77610195-kube-api-access-7bmrj\") pod \"odh-model-controller-696fc77849-c9c6b\" (UID: \"fd326d61-80e0-44e8-be8b-ff6c77610195\") " pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:09.501490 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.501463    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd326d61-80e0-44e8-be8b-ff6c77610195-cert\") pod \"odh-model-controller-696fc77849-c9c6b\" (UID: \"fd326d61-80e0-44e8-be8b-ff6c77610195\") " pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:09.501626 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.501523    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7bmrj\" (UniqueName: \"kubernetes.io/projected/fd326d61-80e0-44e8-be8b-ff6c77610195-kube-api-access-7bmrj\") pod \"odh-model-controller-696fc77849-c9c6b\" (UID: \"fd326d61-80e0-44e8-be8b-ff6c77610195\") " pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:09.503974 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.503949    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fd326d61-80e0-44e8-be8b-ff6c77610195-cert\") pod \"odh-model-controller-696fc77849-c9c6b\" (UID: \"fd326d61-80e0-44e8-be8b-ff6c77610195\") " pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:09.510437 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.510414    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bmrj\" (UniqueName: \"kubernetes.io/projected/fd326d61-80e0-44e8-be8b-ff6c77610195-kube-api-access-7bmrj\") pod \"odh-model-controller-696fc77849-c9c6b\" (UID: \"fd326d61-80e0-44e8-be8b-ff6c77610195\") " pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:09.601507 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.601397    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:09.740026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.739997    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-696fc77849-c9c6b"]
May 25 11:24:09.743431 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:24:09.743402    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd326d61_80e0_44e8_be8b_ff6c77610195.slice/crio-6fcf9a15b57f4e7db6fca9ef6a2ea18448fed90a5b3de378c73db1f17ae11575 WatchSource:0}: Error finding container 6fcf9a15b57f4e7db6fca9ef6a2ea18448fed90a5b3de378c73db1f17ae11575: Status 404 returned error can't find the container with id 6fcf9a15b57f4e7db6fca9ef6a2ea18448fed90a5b3de378c73db1f17ae11575
May 25 11:24:09.744796 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:09.744776    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:24:10.257326 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:10.257283    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-c9c6b" event={"ID":"fd326d61-80e0-44e8-be8b-ff6c77610195","Type":"ContainerStarted","Data":"6fcf9a15b57f4e7db6fca9ef6a2ea18448fed90a5b3de378c73db1f17ae11575"}
May 25 11:24:13.267859 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:13.267826    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-696fc77849-c9c6b" event={"ID":"fd326d61-80e0-44e8-be8b-ff6c77610195","Type":"ContainerStarted","Data":"fc614ceea0c456d3fd8082f7abd6e1cba59ef51306fe355e91d1bdb875fd4eb6"}
May 25 11:24:13.268250 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:13.267933    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:13.303818 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:13.303774    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/odh-model-controller-696fc77849-c9c6b" podStartSLOduration=1.692994527 podStartE2EDuration="4.303762071s" podCreationTimestamp="2026-05-25 11:24:09 +0000 UTC" firstStartedPulling="2026-05-25 11:24:09.744901698 +0000 UTC m=+398.375062485" lastFinishedPulling="2026-05-25 11:24:12.355669229 +0000 UTC m=+400.985830029" observedRunningTime="2026-05-25 11:24:13.302725605 +0000 UTC m=+401.932886416" watchObservedRunningTime="2026-05-25 11:24:13.303762071 +0000 UTC m=+401.933922880"
May 25 11:24:24.272318 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:24.272289    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-696fc77849-c9c6b"
May 25 11:24:25.170316 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.170284    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-n26fb"]
May 25 11:24:25.173568 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.173554    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-n26fb"
May 25 11:24:25.177993 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.177973    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
May 25 11:24:25.178116 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.178025    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-5qhvr\""
May 25 11:24:25.185137 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.185115    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-n26fb"]
May 25 11:24:25.323926 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.323899    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcbgb\" (UniqueName: \"kubernetes.io/projected/5e991ab6-5bc2-44fc-b170-841217759daf-kube-api-access-jcbgb\") pod \"s3-init-n26fb\" (UID: \"5e991ab6-5bc2-44fc-b170-841217759daf\") " pod="kserve/s3-init-n26fb"
May 25 11:24:25.425181 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.425161    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jcbgb\" (UniqueName: \"kubernetes.io/projected/5e991ab6-5bc2-44fc-b170-841217759daf-kube-api-access-jcbgb\") pod \"s3-init-n26fb\" (UID: \"5e991ab6-5bc2-44fc-b170-841217759daf\") " pod="kserve/s3-init-n26fb"
May 25 11:24:25.436856 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.436831    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcbgb\" (UniqueName: \"kubernetes.io/projected/5e991ab6-5bc2-44fc-b170-841217759daf-kube-api-access-jcbgb\") pod \"s3-init-n26fb\" (UID: \"5e991ab6-5bc2-44fc-b170-841217759daf\") " pod="kserve/s3-init-n26fb"
May 25 11:24:25.494309 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.494283    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-n26fb"
May 25 11:24:25.623308 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:25.623285    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-n26fb"]
May 25 11:24:25.625818 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:24:25.625780    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e991ab6_5bc2_44fc_b170_841217759daf.slice/crio-dba748301424b4a046336349410226315f3e71bd4e0d7fb33e2f5928692e13bb WatchSource:0}: Error finding container dba748301424b4a046336349410226315f3e71bd4e0d7fb33e2f5928692e13bb: Status 404 returned error can't find the container with id dba748301424b4a046336349410226315f3e71bd4e0d7fb33e2f5928692e13bb
May 25 11:24:26.309152 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:26.309110    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-n26fb" event={"ID":"5e991ab6-5bc2-44fc-b170-841217759daf","Type":"ContainerStarted","Data":"dba748301424b4a046336349410226315f3e71bd4e0d7fb33e2f5928692e13bb"}
May 25 11:24:30.324036 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:30.324009    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-n26fb" event={"ID":"5e991ab6-5bc2-44fc-b170-841217759daf","Type":"ContainerStarted","Data":"a274c0e2e001e45378d3ca9775e3c785bfbf15d5d87fc19340c509978fd0dce9"}
May 25 11:24:30.338111 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:30.338054    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-n26fb" podStartSLOduration=0.73011246 podStartE2EDuration="5.338036126s" podCreationTimestamp="2026-05-25 11:24:25 +0000 UTC" firstStartedPulling="2026-05-25 11:24:25.627524045 +0000 UTC m=+414.257684834" lastFinishedPulling="2026-05-25 11:24:30.235447697 +0000 UTC m=+418.865608500" observedRunningTime="2026-05-25 11:24:30.336514492 +0000 UTC m=+418.966675303" watchObservedRunningTime="2026-05-25 11:24:30.338036126 +0000 UTC m=+418.968196942"
May 25 11:24:33.333474 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:33.333444    2556 generic.go:358] "Generic (PLEG): container finished" podID="5e991ab6-5bc2-44fc-b170-841217759daf" containerID="a274c0e2e001e45378d3ca9775e3c785bfbf15d5d87fc19340c509978fd0dce9" exitCode=0
May 25 11:24:33.333767 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:33.333517    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-n26fb" event={"ID":"5e991ab6-5bc2-44fc-b170-841217759daf","Type":"ContainerDied","Data":"a274c0e2e001e45378d3ca9775e3c785bfbf15d5d87fc19340c509978fd0dce9"}
May 25 11:24:34.460078 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:34.460054    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-n26fb"
May 25 11:24:34.606220 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:34.606157    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jcbgb\" (UniqueName: \"kubernetes.io/projected/5e991ab6-5bc2-44fc-b170-841217759daf-kube-api-access-jcbgb\") pod \"5e991ab6-5bc2-44fc-b170-841217759daf\" (UID: \"5e991ab6-5bc2-44fc-b170-841217759daf\") "
May 25 11:24:34.608184 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:34.608162    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e991ab6-5bc2-44fc-b170-841217759daf-kube-api-access-jcbgb" (OuterVolumeSpecName: "kube-api-access-jcbgb") pod "5e991ab6-5bc2-44fc-b170-841217759daf" (UID: "5e991ab6-5bc2-44fc-b170-841217759daf"). InnerVolumeSpecName "kube-api-access-jcbgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:24:34.706653 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:34.706631    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcbgb\" (UniqueName: \"kubernetes.io/projected/5e991ab6-5bc2-44fc-b170-841217759daf-kube-api-access-jcbgb\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:24:35.341131 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:35.341105    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-n26fb"
May 25 11:24:35.341292 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:35.341106    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-n26fb" event={"ID":"5e991ab6-5bc2-44fc-b170-841217759daf","Type":"ContainerDied","Data":"dba748301424b4a046336349410226315f3e71bd4e0d7fb33e2f5928692e13bb"}
May 25 11:24:35.341292 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:35.341217    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dba748301424b4a046336349410226315f3e71bd4e0d7fb33e2f5928692e13bb"
May 25 11:24:47.417311 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.417229    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"]
May 25 11:24:47.417773 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.417595    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e991ab6-5bc2-44fc-b170-841217759daf" containerName="s3-init"
May 25 11:24:47.417773 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.417607    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e991ab6-5bc2-44fc-b170-841217759daf" containerName="s3-init"
May 25 11:24:47.417773 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.417679    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="5e991ab6-5bc2-44fc-b170-841217759daf" containerName="s3-init"
May 25 11:24:47.422210 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.422194    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.430004 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.429984    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-7006d-predictor-serving-cert\""
May 25 11:24:47.430124 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.430036    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\""
May 25 11:24:47.431116 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.431092    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
May 25 11:24:47.431212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.431092    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qhlr\""
May 25 11:24:47.432180 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.432021    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
May 25 11:24:47.433256 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.433235    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"]
May 25 11:24:47.513249 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.513215    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74d7eab-07de-4a1b-8e09-d288351d088c-isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.513409 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.513287    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74d7eab-07de-4a1b-8e09-d288351d088c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.513409 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.513346    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74d7eab-07de-4a1b-8e09-d288351d088c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.513409 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.513400    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fqb2\" (UniqueName: \"kubernetes.io/projected/d74d7eab-07de-4a1b-8e09-d288351d088c-kube-api-access-5fqb2\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.613878 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.613844    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74d7eab-07de-4a1b-8e09-d288351d088c-isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.614007 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.613906    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74d7eab-07de-4a1b-8e09-d288351d088c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.614007 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.613951    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74d7eab-07de-4a1b-8e09-d288351d088c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.614112 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.614008    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5fqb2\" (UniqueName: \"kubernetes.io/projected/d74d7eab-07de-4a1b-8e09-d288351d088c-kube-api-access-5fqb2\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.614438 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.614416    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74d7eab-07de-4a1b-8e09-d288351d088c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.614763 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.614744    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74d7eab-07de-4a1b-8e09-d288351d088c-isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.616288 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.616272    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74d7eab-07de-4a1b-8e09-d288351d088c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.621805 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.621786    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fqb2\" (UniqueName: \"kubernetes.io/projected/d74d7eab-07de-4a1b-8e09-d288351d088c-kube-api-access-5fqb2\") pod \"isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.733418 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.733393    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:24:47.862060 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:47.862034    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"]
May 25 11:24:47.864422 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:24:47.864395    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74d7eab_07de_4a1b_8e09_d288351d088c.slice/crio-6abdc260acdd7bf85312cfffc54ffd6501fff7a471ba20a5ddddbee5def3fa12 WatchSource:0}: Error finding container 6abdc260acdd7bf85312cfffc54ffd6501fff7a471ba20a5ddddbee5def3fa12: Status 404 returned error can't find the container with id 6abdc260acdd7bf85312cfffc54ffd6501fff7a471ba20a5ddddbee5def3fa12
May 25 11:24:48.380357 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:48.380321    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerStarted","Data":"6abdc260acdd7bf85312cfffc54ffd6501fff7a471ba20a5ddddbee5def3fa12"}
May 25 11:24:52.395162 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:52.395124    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerStarted","Data":"60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d"}
May 25 11:24:55.411277 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:55.411244    2556 generic.go:358] "Generic (PLEG): container finished" podID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerID="60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d" exitCode=0
May 25 11:24:55.411742 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:24:55.411317    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerDied","Data":"60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d"}
May 25 11:25:09.463677 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:09.463624    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerStarted","Data":"8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56"}
May 25 11:25:11.472358 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:11.472314    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerStarted","Data":"d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479"}
May 25 11:25:14.485253 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:14.485221    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerStarted","Data":"5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9"}
May 25 11:25:14.485620 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:14.485508    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:25:14.485620 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:14.485609    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:25:14.486620 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:14.486591    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:25:14.506883 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:14.506840    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podStartSLOduration=1.5014961919999998 podStartE2EDuration="27.506829271s" podCreationTimestamp="2026-05-25 11:24:47 +0000 UTC" firstStartedPulling="2026-05-25 11:24:47.866184119 +0000 UTC m=+436.496344910" lastFinishedPulling="2026-05-25 11:25:13.871517188 +0000 UTC m=+462.501677989" observedRunningTime="2026-05-25 11:25:14.505393649 +0000 UTC m=+463.135554460" watchObservedRunningTime="2026-05-25 11:25:14.506829271 +0000 UTC m=+463.136990080"
May 25 11:25:15.488185 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:15.488128    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:25:15.488554 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:15.488157    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:25:15.488986 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:15.488963    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:25:16.491178 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:16.491130    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:25:16.491624 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:16.491452    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:25:16.494677 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:16.494654    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:25:17.494662 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:17.494626    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:25:17.495076 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:17.494850    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:25:27.494885 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:27.494829    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:25:27.495312 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:27.495231    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:25:37.494781 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:37.494731    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:25:37.495289 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:37.495265    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:25:47.495416 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:47.495362    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:25:47.495871 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:47.495833    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:25:57.495564 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:57.495519    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:25:57.496023 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:25:57.496000    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:07.494592 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:07.494549    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:26:07.495015 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:07.494987    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:17.495729 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:17.495646    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:26:17.496135 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:17.496116    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:26:32.448873 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.448844    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"]
May 25 11:26:32.449325 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.449189    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" containerID="cri-o://8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56" gracePeriod=30
May 25 11:26:32.449325 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.449215    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy" containerID="cri-o://d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479" gracePeriod=30
May 25 11:26:32.449325 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.449240    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" containerID="cri-o://5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9" gracePeriod=30
May 25 11:26:32.570081 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.570056    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"]
May 25 11:26:32.573392 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.573375    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.575773 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.575754    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\""
May 25 11:26:32.575908 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.575879    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-9ad8c-predictor-serving-cert\""
May 25 11:26:32.585023 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.584981    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"]
May 25 11:26:32.661284 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.661252    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"]
May 25 11:26:32.664705 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.664686    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.665877 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.665849    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/128bacb7-b517-4455-b06f-2ac7ee595cf9-isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.665977 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.665910    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/128bacb7-b517-4455-b06f-2ac7ee595cf9-proxy-tls\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.666058 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.666000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128bacb7-b517-4455-b06f-2ac7ee595cf9-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.666058 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.666042    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj7br\" (UniqueName: \"kubernetes.io/projected/128bacb7-b517-4455-b06f-2ac7ee595cf9-kube-api-access-cj7br\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.667253 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.667235    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-9ad8c-predictor-serving-cert\""
May 25 11:26:32.667367 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.667350    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\""
May 25 11:26:32.678193 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.678170    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"]
May 25 11:26:32.713559 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.713473    2556 generic.go:358] "Generic (PLEG): container finished" podID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerID="d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479" exitCode=2
May 25 11:26:32.713559 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.713546    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerDied","Data":"d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479"}
May 25 11:26:32.767089 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767064    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/128bacb7-b517-4455-b06f-2ac7ee595cf9-proxy-tls\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.767206 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767104    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128bacb7-b517-4455-b06f-2ac7ee595cf9-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.767206 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50bfa7dd-de16-483f-b499-710061a40640-proxy-tls\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.767206 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767145    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50bfa7dd-de16-483f-b499-710061a40640-isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.767206 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767166    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cj7br\" (UniqueName: \"kubernetes.io/projected/128bacb7-b517-4455-b06f-2ac7ee595cf9-kube-api-access-cj7br\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.767206 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767184    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nwkm\" (UniqueName: \"kubernetes.io/projected/50bfa7dd-de16-483f-b499-710061a40640-kube-api-access-5nwkm\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.767453 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767239    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50bfa7dd-de16-483f-b499-710061a40640-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.767453 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767354    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/128bacb7-b517-4455-b06f-2ac7ee595cf9-isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.767585 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767465    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128bacb7-b517-4455-b06f-2ac7ee595cf9-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.767930 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.767913    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/128bacb7-b517-4455-b06f-2ac7ee595cf9-isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.769512 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.769473    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/128bacb7-b517-4455-b06f-2ac7ee595cf9-proxy-tls\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.775475 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.775454    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj7br\" (UniqueName: \"kubernetes.io/projected/128bacb7-b517-4455-b06f-2ac7ee595cf9-kube-api-access-cj7br\") pod \"isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.868210 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.868182    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50bfa7dd-de16-483f-b499-710061a40640-proxy-tls\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.868346 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.868218    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50bfa7dd-de16-483f-b499-710061a40640-isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.868346 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.868242    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5nwkm\" (UniqueName: \"kubernetes.io/projected/50bfa7dd-de16-483f-b499-710061a40640-kube-api-access-5nwkm\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.868346 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.868262    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50bfa7dd-de16-483f-b499-710061a40640-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.868703 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.868681    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50bfa7dd-de16-483f-b499-710061a40640-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.868901 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.868883    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50bfa7dd-de16-483f-b499-710061a40640-isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.870472 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.870456    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50bfa7dd-de16-483f-b499-710061a40640-proxy-tls\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.876818 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.876786    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nwkm\" (UniqueName: \"kubernetes.io/projected/50bfa7dd-de16-483f-b499-710061a40640-kube-api-access-5nwkm\") pod \"isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:32.885551 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.885523    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:32.975225 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:32.975159    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:33.013181 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:33.012922    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"]
May 25 11:26:33.015553 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:26:33.015517    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128bacb7_b517_4455_b06f_2ac7ee595cf9.slice/crio-9bf8d40c737d7b2e2a038537bbe37e1127be4cd3a4c970bd026b4e999973e97b WatchSource:0}: Error finding container 9bf8d40c737d7b2e2a038537bbe37e1127be4cd3a4c970bd026b4e999973e97b: Status 404 returned error can't find the container with id 9bf8d40c737d7b2e2a038537bbe37e1127be4cd3a4c970bd026b4e999973e97b
May 25 11:26:33.108241 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:33.108205    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"]
May 25 11:26:33.110863 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:26:33.110828    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bfa7dd_de16_483f_b499_710061a40640.slice/crio-d2bb482404c4f6f9a9176d887e2d0a416f1d7a011e254b7665ab09215b6fb42d WatchSource:0}: Error finding container d2bb482404c4f6f9a9176d887e2d0a416f1d7a011e254b7665ab09215b6fb42d: Status 404 returned error can't find the container with id d2bb482404c4f6f9a9176d887e2d0a416f1d7a011e254b7665ab09215b6fb42d
May 25 11:26:33.717702 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:33.717660    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" event={"ID":"50bfa7dd-de16-483f-b499-710061a40640","Type":"ContainerStarted","Data":"16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d"}
May 25 11:26:33.717702 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:33.717706    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" event={"ID":"50bfa7dd-de16-483f-b499-710061a40640","Type":"ContainerStarted","Data":"d2bb482404c4f6f9a9176d887e2d0a416f1d7a011e254b7665ab09215b6fb42d"}
May 25 11:26:33.719092 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:33.719065    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" event={"ID":"128bacb7-b517-4455-b06f-2ac7ee595cf9","Type":"ContainerStarted","Data":"e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5"}
May 25 11:26:33.719228 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:33.719096    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" event={"ID":"128bacb7-b517-4455-b06f-2ac7ee595cf9","Type":"ContainerStarted","Data":"9bf8d40c737d7b2e2a038537bbe37e1127be4cd3a4c970bd026b4e999973e97b"}
May 25 11:26:36.491623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:36.491589    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
May 25 11:26:36.729605 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:36.729567    2556 generic.go:358] "Generic (PLEG): container finished" podID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerID="8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56" exitCode=0
May 25 11:26:36.729605 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:36.729606    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerDied","Data":"8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56"}
May 25 11:26:37.495477 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:37.495443    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:26:37.497044 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:37.497015    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:37.733546 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:37.733523    2556 generic.go:358] "Generic (PLEG): container finished" podID="50bfa7dd-de16-483f-b499-710061a40640" containerID="16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d" exitCode=0
May 25 11:26:37.733754 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:37.733590    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" event={"ID":"50bfa7dd-de16-483f-b499-710061a40640","Type":"ContainerDied","Data":"16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d"}
May 25 11:26:37.734977 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:37.734956    2556 generic.go:358] "Generic (PLEG): container finished" podID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerID="e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5" exitCode=0
May 25 11:26:37.735072 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:37.735025    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" event={"ID":"128bacb7-b517-4455-b06f-2ac7ee595cf9","Type":"ContainerDied","Data":"e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5"}
May 25 11:26:38.740779 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:38.740725    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" event={"ID":"128bacb7-b517-4455-b06f-2ac7ee595cf9","Type":"ContainerStarted","Data":"ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40"}
May 25 11:26:38.740779 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:38.740781    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" event={"ID":"128bacb7-b517-4455-b06f-2ac7ee595cf9","Type":"ContainerStarted","Data":"36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062"}
May 25 11:26:38.741260 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:38.741017    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:38.764339 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:38.764289    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podStartSLOduration=6.764272966 podStartE2EDuration="6.764272966s" podCreationTimestamp="2026-05-25 11:26:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:26:38.761591555 +0000 UTC m=+547.391752368" watchObservedRunningTime="2026-05-25 11:26:38.764272966 +0000 UTC m=+547.394433777"
May 25 11:26:39.746790 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:39.746740    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:39.748320 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:39.748285    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 11:26:40.749648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:40.749602    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 11:26:41.491739 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:41.491703    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
May 25 11:26:45.756030 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:45.755994    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:26:45.756699 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:45.756660    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 11:26:46.491946 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:46.491904    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
May 25 11:26:46.492113 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:46.492061    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:26:47.495090 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:47.495041    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:26:47.496825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:47.496794    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:51.491592 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:51.491536    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
May 25 11:26:55.756782 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:55.756738    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 11:26:56.491358 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:56.491313    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
May 25 11:26:56.805432 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:56.805352    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" event={"ID":"50bfa7dd-de16-483f-b499-710061a40640","Type":"ContainerStarted","Data":"b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023"}
May 25 11:26:56.805432 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:56.805395    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" event={"ID":"50bfa7dd-de16-483f-b499-710061a40640","Type":"ContainerStarted","Data":"8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616"}
May 25 11:26:56.805828 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:56.805623    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:56.832890 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:56.829269    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podStartSLOduration=6.503166742 podStartE2EDuration="24.829250811s" podCreationTimestamp="2026-05-25 11:26:32 +0000 UTC" firstStartedPulling="2026-05-25 11:26:37.734891136 +0000 UTC m=+546.365051925" lastFinishedPulling="2026-05-25 11:26:56.060975194 +0000 UTC m=+564.691135994" observedRunningTime="2026-05-25 11:26:56.826069745 +0000 UTC m=+565.456230566" watchObservedRunningTime="2026-05-25 11:26:56.829250811 +0000 UTC m=+565.459411626"
May 25 11:26:57.495258 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:57.495219    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
May 25 11:26:57.495420 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:57.495374    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:26:57.496800 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:57.496775    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:26:57.496892 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:57.496881    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:26:57.808857 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:57.808781    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:26:57.810004 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:57.809982    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 11:26:58.811520 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:26:58.811462    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 11:27:01.491250 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:01.491211    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
May 25 11:27:02.587131 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.587107    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:27:02.632983 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.632956    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fqb2\" (UniqueName: \"kubernetes.io/projected/d74d7eab-07de-4a1b-8e09-d288351d088c-kube-api-access-5fqb2\") pod \"d74d7eab-07de-4a1b-8e09-d288351d088c\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") "
May 25 11:27:02.633152 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.633008    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74d7eab-07de-4a1b-8e09-d288351d088c-isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\") pod \"d74d7eab-07de-4a1b-8e09-d288351d088c\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") "
May 25 11:27:02.633152 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.633071    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74d7eab-07de-4a1b-8e09-d288351d088c-kserve-provision-location\") pod \"d74d7eab-07de-4a1b-8e09-d288351d088c\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") "
May 25 11:27:02.633152 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.633127    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74d7eab-07de-4a1b-8e09-d288351d088c-proxy-tls\") pod \"d74d7eab-07de-4a1b-8e09-d288351d088c\" (UID: \"d74d7eab-07de-4a1b-8e09-d288351d088c\") "
May 25 11:27:02.633440 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.633413    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74d7eab-07de-4a1b-8e09-d288351d088c-isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config") pod "d74d7eab-07de-4a1b-8e09-d288351d088c" (UID: "d74d7eab-07de-4a1b-8e09-d288351d088c"). InnerVolumeSpecName "isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:27:02.633440 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.633420    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74d7eab-07de-4a1b-8e09-d288351d088c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d74d7eab-07de-4a1b-8e09-d288351d088c" (UID: "d74d7eab-07de-4a1b-8e09-d288351d088c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:27:02.635445 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.635421    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74d7eab-07de-4a1b-8e09-d288351d088c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d74d7eab-07de-4a1b-8e09-d288351d088c" (UID: "d74d7eab-07de-4a1b-8e09-d288351d088c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:27:02.635654 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.635638    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74d7eab-07de-4a1b-8e09-d288351d088c-kube-api-access-5fqb2" (OuterVolumeSpecName: "kube-api-access-5fqb2") pod "d74d7eab-07de-4a1b-8e09-d288351d088c" (UID: "d74d7eab-07de-4a1b-8e09-d288351d088c"). InnerVolumeSpecName "kube-api-access-5fqb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:27:02.733891 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.733819    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74d7eab-07de-4a1b-8e09-d288351d088c-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:27:02.733891 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.733848    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74d7eab-07de-4a1b-8e09-d288351d088c-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:27:02.733891 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.733859    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fqb2\" (UniqueName: \"kubernetes.io/projected/d74d7eab-07de-4a1b-8e09-d288351d088c-kube-api-access-5fqb2\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:27:02.733891 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.733869    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74d7eab-07de-4a1b-8e09-d288351d088c-isvc-raw-sklearn-batcher-7006d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:27:02.824781 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.824748    2556 generic.go:358] "Generic (PLEG): container finished" podID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerID="5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9" exitCode=0
May 25 11:27:02.824928 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.824837    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"
May 25 11:27:02.824928 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.824835    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerDied","Data":"5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9"}
May 25 11:27:02.824928 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.824880    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd" event={"ID":"d74d7eab-07de-4a1b-8e09-d288351d088c","Type":"ContainerDied","Data":"6abdc260acdd7bf85312cfffc54ffd6501fff7a471ba20a5ddddbee5def3fa12"}
May 25 11:27:02.824928 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.824901    2556 scope.go:117] "RemoveContainer" containerID="5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9"
May 25 11:27:02.833522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.833475    2556 scope.go:117] "RemoveContainer" containerID="d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479"
May 25 11:27:02.841203 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.841184    2556 scope.go:117] "RemoveContainer" containerID="8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56"
May 25 11:27:02.848230 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.848213    2556 scope.go:117] "RemoveContainer" containerID="60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d"
May 25 11:27:02.852336 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.852300    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"]
May 25 11:27:02.855225 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.855210    2556 scope.go:117] "RemoveContainer" containerID="5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9"
May 25 11:27:02.855493 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:27:02.855462    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9\": container with ID starting with 5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9 not found: ID does not exist" containerID="5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9"
May 25 11:27:02.855559 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.855507    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9"} err="failed to get container status \"5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9\": rpc error: code = NotFound desc = could not find container \"5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9\": container with ID starting with 5e502c7f3aa4dbf5699a7aca9a4319a7dcca302e95c30df3927421c35e7bdec9 not found: ID does not exist"
May 25 11:27:02.855559 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.855533    2556 scope.go:117] "RemoveContainer" containerID="d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479"
May 25 11:27:02.855795 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:27:02.855778    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479\": container with ID starting with d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479 not found: ID does not exist" containerID="d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479"
May 25 11:27:02.855875 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.855800    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479"} err="failed to get container status \"d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479\": rpc error: code = NotFound desc = could not find container \"d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479\": container with ID starting with d1e7a89e1fb279cd400e41b393c4b906c55fd60cc4d65e718e009db797e42479 not found: ID does not exist"
May 25 11:27:02.855875 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.855815    2556 scope.go:117] "RemoveContainer" containerID="8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56"
May 25 11:27:02.856136 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:27:02.856067    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56\": container with ID starting with 8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56 not found: ID does not exist" containerID="8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56"
May 25 11:27:02.856215 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.856192    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56"} err="failed to get container status \"8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56\": rpc error: code = NotFound desc = could not find container \"8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56\": container with ID starting with 8572d2a171156a295626776173e47dff4589422dd2e4ae30d034a0da795bef56 not found: ID does not exist"
May 25 11:27:02.856320 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.856221    2556 scope.go:117] "RemoveContainer" containerID="60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d"
May 25 11:27:02.856701 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:27:02.856667    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d\": container with ID starting with 60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d not found: ID does not exist" containerID="60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d"
May 25 11:27:02.856790 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.856701    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d"} err="failed to get container status \"60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d\": rpc error: code = NotFound desc = could not find container \"60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d\": container with ID starting with 60f0e08109b5ea5e72d437719f522eb7f725aa030270316e84702697f29d899d not found: ID does not exist"
May 25 11:27:02.858561 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:02.858543    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-7006d-predictor-74696f9d9f-bpgpd"]
May 25 11:27:03.815372 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:03.815338    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:27:03.815982 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:03.815943    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 11:27:03.982676 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:03.982639    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" path="/var/lib/kubelet/pods/d74d7eab-07de-4a1b-8e09-d288351d088c/volumes"
May 25 11:27:05.756843 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:05.756806    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 11:27:13.816001 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:13.815959    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 11:27:15.757002 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:15.756957    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 11:27:23.816744 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:23.816702    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 11:27:25.756538 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:25.756500    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 11:27:31.881499 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:31.881452    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:27:31.882166 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:31.882144    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:27:33.816338 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:33.816297    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 11:27:35.757248 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:35.757211    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 11:27:43.816309 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:43.816226    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 11:27:45.757602 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:45.757575    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:27:53.816535 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:27:53.816505    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:28:12.786328 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.786289    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"]
May 25 11:28:12.786760 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.786613    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" containerID="cri-o://36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062" gracePeriod=30
May 25 11:28:12.786760 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.786664    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kube-rbac-proxy" containerID="cri-o://ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40" gracePeriod=30
May 25 11:28:12.874211 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874182    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"]
May 25 11:28:12.874597 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874584    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="storage-initializer"
May 25 11:28:12.874646 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874599    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="storage-initializer"
May 25 11:28:12.874646 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874610    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container"
May 25 11:28:12.874646 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874616    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container"
May 25 11:28:12.874646 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874626    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent"
May 25 11:28:12.874646 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874632    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent"
May 25 11:28:12.874646 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874641    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy"
May 25 11:28:12.874839 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874649    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy"
May 25 11:28:12.874839 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874702    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="agent"
May 25 11:28:12.874839 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874710    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kserve-container"
May 25 11:28:12.874839 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.874722    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="d74d7eab-07de-4a1b-8e09-d288351d088c" containerName="kube-rbac-proxy"
May 25 11:28:12.877237 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.877223    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:12.879720 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.879702    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-6729d-predictor-serving-cert\""
May 25 11:28:12.880024 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.880006    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\""
May 25 11:28:12.889621 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.889598    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"]
May 25 11:28:12.934166 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.934138    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"]
May 25 11:28:12.934543 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.934496    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container" containerID="cri-o://8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616" gracePeriod=30
May 25 11:28:12.934652 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.934524    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kube-rbac-proxy" containerID="cri-o://b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023" gracePeriod=30
May 25 11:28:12.970641 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.970612    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"]
May 25 11:28:12.973574 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.973552    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:12.976081 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.976056    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-6729d-predictor-serving-cert\""
May 25 11:28:12.976404 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.976092    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\""
May 25 11:28:12.978373 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.978347    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:12.978522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.978458    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86xx5\" (UniqueName: \"kubernetes.io/projected/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kube-api-access-86xx5\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:12.978573 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.978529    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:12.978573 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.978552    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:12.985856 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:12.985835    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"]
May 25 11:28:13.052293 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.052266    2556 generic.go:358] "Generic (PLEG): container finished" podID="50bfa7dd-de16-483f-b499-710061a40640" containerID="b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023" exitCode=2
May 25 11:28:13.052426 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.052336    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" event={"ID":"50bfa7dd-de16-483f-b499-710061a40640","Type":"ContainerDied","Data":"b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023"}
May 25 11:28:13.054129 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.054107    2556 generic.go:358] "Generic (PLEG): container finished" podID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerID="ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40" exitCode=2
May 25 11:28:13.054216 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.054184    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" event={"ID":"128bacb7-b517-4455-b06f-2ac7ee595cf9","Type":"ContainerDied","Data":"ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40"}
May 25 11:28:13.079686 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.079659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48trw\" (UniqueName: \"kubernetes.io/projected/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kube-api-access-48trw\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.079782 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.079698    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-86xx5\" (UniqueName: \"kubernetes.io/projected/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kube-api-access-86xx5\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.079782 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.079731    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.079782 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.079760    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3ccca0-1f20-4bbf-8543-5cb6643d6530-isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.079944 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.079793    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.079944 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.079892    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.079944 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:28:13.079904    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-serving-cert: secret "isvc-sklearn-graph-raw-hpa-6729d-predictor-serving-cert" not found
May 25 11:28:13.080079 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.079970    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3ccca0-1f20-4bbf-8543-5cb6643d6530-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.080079 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:28:13.079982    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-proxy-tls podName:3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b nodeName:}" failed. No retries permitted until 2026-05-25 11:28:13.579958428 +0000 UTC m=+642.210119230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-proxy-tls") pod "isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" (UID: "3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b") : secret "isvc-sklearn-graph-raw-hpa-6729d-predictor-serving-cert" not found
May 25 11:28:13.080079 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.080048    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.080226 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.080049    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.080526 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.080507    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.088523 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.088501    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86xx5\" (UniqueName: \"kubernetes.io/projected/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kube-api-access-86xx5\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.180873 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.180839    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.181037 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.180880    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3ccca0-1f20-4bbf-8543-5cb6643d6530-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.181037 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.180937    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-48trw\" (UniqueName: \"kubernetes.io/projected/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kube-api-access-48trw\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.181037 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.180974    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3ccca0-1f20-4bbf-8543-5cb6643d6530-isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.181306 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.181282    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.181674 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.181652    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3ccca0-1f20-4bbf-8543-5cb6643d6530-isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.183391 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.183362    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3ccca0-1f20-4bbf-8543-5cb6643d6530-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.189710 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.189687    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48trw\" (UniqueName: \"kubernetes.io/projected/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kube-api-access-48trw\") pod \"isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.284210 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.284173    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:13.418759 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.418724    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"]
May 25 11:28:13.422413 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:28:13.422388    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3ccca0_1f20_4bbf_8543_5cb6643d6530.slice/crio-28008799e39011b4f0bc2638700308a694eb6e6829bcdf0a07db1d1b3e4f6831 WatchSource:0}: Error finding container 28008799e39011b4f0bc2638700308a694eb6e6829bcdf0a07db1d1b3e4f6831: Status 404 returned error can't find the container with id 28008799e39011b4f0bc2638700308a694eb6e6829bcdf0a07db1d1b3e4f6831
May 25 11:28:13.585178 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.585103    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.587373 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.587354    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.787112 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.787073    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:13.812094 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.812057    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
May 25 11:28:13.816630 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.816601    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
May 25 11:28:13.952773 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:13.952744    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"]
May 25 11:28:13.953397 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:28:13.953371    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce9ab66_f3ad_4ac9_9d78_29cc7ce36a6b.slice/crio-1d666c082b115c148bf0c2f74b39628794a1c2aecc574c5e19827c9a08273e70 WatchSource:0}: Error finding container 1d666c082b115c148bf0c2f74b39628794a1c2aecc574c5e19827c9a08273e70: Status 404 returned error can't find the container with id 1d666c082b115c148bf0c2f74b39628794a1c2aecc574c5e19827c9a08273e70
May 25 11:28:14.059048 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:14.059010    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" event={"ID":"de3ccca0-1f20-4bbf-8543-5cb6643d6530","Type":"ContainerStarted","Data":"a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f"}
May 25 11:28:14.059208 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:14.059055    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" event={"ID":"de3ccca0-1f20-4bbf-8543-5cb6643d6530","Type":"ContainerStarted","Data":"28008799e39011b4f0bc2638700308a694eb6e6829bcdf0a07db1d1b3e4f6831"}
May 25 11:28:14.060526 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:14.060501    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" event={"ID":"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b","Type":"ContainerStarted","Data":"b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0"}
May 25 11:28:14.060631 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:14.060534    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" event={"ID":"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b","Type":"ContainerStarted","Data":"1d666c082b115c148bf0c2f74b39628794a1c2aecc574c5e19827c9a08273e70"}
May 25 11:28:15.750200 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:15.750162    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
May 25 11:28:15.756703 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:15.756681    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
May 25 11:28:16.674975 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.674946    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:28:16.815354 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.815269    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nwkm\" (UniqueName: \"kubernetes.io/projected/50bfa7dd-de16-483f-b499-710061a40640-kube-api-access-5nwkm\") pod \"50bfa7dd-de16-483f-b499-710061a40640\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") "
May 25 11:28:16.815354 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.815323    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50bfa7dd-de16-483f-b499-710061a40640-proxy-tls\") pod \"50bfa7dd-de16-483f-b499-710061a40640\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") "
May 25 11:28:16.815354 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.815355    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50bfa7dd-de16-483f-b499-710061a40640-isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") pod \"50bfa7dd-de16-483f-b499-710061a40640\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") "
May 25 11:28:16.815818 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.815394    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50bfa7dd-de16-483f-b499-710061a40640-kserve-provision-location\") pod \"50bfa7dd-de16-483f-b499-710061a40640\" (UID: \"50bfa7dd-de16-483f-b499-710061a40640\") "
May 25 11:28:16.815818 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.815770    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bfa7dd-de16-483f-b499-710061a40640-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50bfa7dd-de16-483f-b499-710061a40640" (UID: "50bfa7dd-de16-483f-b499-710061a40640"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:28:16.815818 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.815783    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50bfa7dd-de16-483f-b499-710061a40640-isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config") pod "50bfa7dd-de16-483f-b499-710061a40640" (UID: "50bfa7dd-de16-483f-b499-710061a40640"). InnerVolumeSpecName "isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:28:16.817387 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.817362    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bfa7dd-de16-483f-b499-710061a40640-kube-api-access-5nwkm" (OuterVolumeSpecName: "kube-api-access-5nwkm") pod "50bfa7dd-de16-483f-b499-710061a40640" (UID: "50bfa7dd-de16-483f-b499-710061a40640"). InnerVolumeSpecName "kube-api-access-5nwkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:28:16.817462 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.817446    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50bfa7dd-de16-483f-b499-710061a40640-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50bfa7dd-de16-483f-b499-710061a40640" (UID: "50bfa7dd-de16-483f-b499-710061a40640"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:28:16.916195 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.916159    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nwkm\" (UniqueName: \"kubernetes.io/projected/50bfa7dd-de16-483f-b499-710061a40640-kube-api-access-5nwkm\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:28:16.916195 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.916192    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50bfa7dd-de16-483f-b499-710061a40640-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:28:16.916446 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.916207    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50bfa7dd-de16-483f-b499-710061a40640-isvc-xgboost-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:28:16.916446 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:16.916221    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50bfa7dd-de16-483f-b499-710061a40640-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:28:17.071049 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.070976    2556 generic.go:358] "Generic (PLEG): container finished" podID="50bfa7dd-de16-483f-b499-710061a40640" containerID="8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616" exitCode=0
May 25 11:28:17.071207 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.071056    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"
May 25 11:28:17.071207 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.071058    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" event={"ID":"50bfa7dd-de16-483f-b499-710061a40640","Type":"ContainerDied","Data":"8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616"}
May 25 11:28:17.071207 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.071104    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn" event={"ID":"50bfa7dd-de16-483f-b499-710061a40640","Type":"ContainerDied","Data":"d2bb482404c4f6f9a9176d887e2d0a416f1d7a011e254b7665ab09215b6fb42d"}
May 25 11:28:17.071207 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.071124    2556 scope.go:117] "RemoveContainer" containerID="b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023"
May 25 11:28:17.079548 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.079528    2556 scope.go:117] "RemoveContainer" containerID="8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616"
May 25 11:28:17.086930 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.086912    2556 scope.go:117] "RemoveContainer" containerID="16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d"
May 25 11:28:17.093655 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.093640    2556 scope.go:117] "RemoveContainer" containerID="b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023"
May 25 11:28:17.093926 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:28:17.093905    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023\": container with ID starting with b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023 not found: ID does not exist" containerID="b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023"
May 25 11:28:17.093990 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.093932    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023"} err="failed to get container status \"b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023\": rpc error: code = NotFound desc = could not find container \"b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023\": container with ID starting with b03c124197d84c3e728432d5fa18b75cbfa87be92350dc19a771728ba737e023 not found: ID does not exist"
May 25 11:28:17.093990 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.093950    2556 scope.go:117] "RemoveContainer" containerID="8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616"
May 25 11:28:17.094176 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:28:17.094159    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616\": container with ID starting with 8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616 not found: ID does not exist" containerID="8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616"
May 25 11:28:17.094221 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.094182    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616"} err="failed to get container status \"8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616\": rpc error: code = NotFound desc = could not find container \"8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616\": container with ID starting with 8aaeb3f90124c0a8da29f83f0d087cfa2fa0b7e69fb9d77e51423751c71a4616 not found: ID does not exist"
May 25 11:28:17.094221 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.094199    2556 scope.go:117] "RemoveContainer" containerID="16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d"
May 25 11:28:17.094448 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:28:17.094432    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d\": container with ID starting with 16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d not found: ID does not exist" containerID="16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d"
May 25 11:28:17.094520 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.094452    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d"} err="failed to get container status \"16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d\": rpc error: code = NotFound desc = could not find container \"16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d\": container with ID starting with 16ac7a8e9419a3ef67fbd6672e56cfd48010553c67131045cc897dbce731489d not found: ID does not exist"
May 25 11:28:17.115337 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.115305    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"]
May 25 11:28:17.122684 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.122662    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-9ad8c-predictor-6d9bd8d997-d4lpn"]
May 25 11:28:17.313959 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.313936    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:28:17.420920 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.420835    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/128bacb7-b517-4455-b06f-2ac7ee595cf9-proxy-tls\") pod \"128bacb7-b517-4455-b06f-2ac7ee595cf9\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") "
May 25 11:28:17.421074 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.420932    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128bacb7-b517-4455-b06f-2ac7ee595cf9-kserve-provision-location\") pod \"128bacb7-b517-4455-b06f-2ac7ee595cf9\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") "
May 25 11:28:17.421074 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.420984    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj7br\" (UniqueName: \"kubernetes.io/projected/128bacb7-b517-4455-b06f-2ac7ee595cf9-kube-api-access-cj7br\") pod \"128bacb7-b517-4455-b06f-2ac7ee595cf9\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") "
May 25 11:28:17.421074 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.421020    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/128bacb7-b517-4455-b06f-2ac7ee595cf9-isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") pod \"128bacb7-b517-4455-b06f-2ac7ee595cf9\" (UID: \"128bacb7-b517-4455-b06f-2ac7ee595cf9\") "
May 25 11:28:17.421267 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.421237    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/128bacb7-b517-4455-b06f-2ac7ee595cf9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "128bacb7-b517-4455-b06f-2ac7ee595cf9" (UID: "128bacb7-b517-4455-b06f-2ac7ee595cf9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:28:17.421476 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.421445    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/128bacb7-b517-4455-b06f-2ac7ee595cf9-isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config") pod "128bacb7-b517-4455-b06f-2ac7ee595cf9" (UID: "128bacb7-b517-4455-b06f-2ac7ee595cf9"). InnerVolumeSpecName "isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:28:17.422991 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.422965    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/128bacb7-b517-4455-b06f-2ac7ee595cf9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "128bacb7-b517-4455-b06f-2ac7ee595cf9" (UID: "128bacb7-b517-4455-b06f-2ac7ee595cf9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:28:17.423070 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.422998    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/128bacb7-b517-4455-b06f-2ac7ee595cf9-kube-api-access-cj7br" (OuterVolumeSpecName: "kube-api-access-cj7br") pod "128bacb7-b517-4455-b06f-2ac7ee595cf9" (UID: "128bacb7-b517-4455-b06f-2ac7ee595cf9"). InnerVolumeSpecName "kube-api-access-cj7br". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:28:17.522628 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.522597    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/128bacb7-b517-4455-b06f-2ac7ee595cf9-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:28:17.522628 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.522625    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/128bacb7-b517-4455-b06f-2ac7ee595cf9-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:28:17.522786 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.522636    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj7br\" (UniqueName: \"kubernetes.io/projected/128bacb7-b517-4455-b06f-2ac7ee595cf9-kube-api-access-cj7br\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:28:17.522786 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.522648    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/128bacb7-b517-4455-b06f-2ac7ee595cf9-isvc-sklearn-graph-raw-9ad8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:28:17.980659 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:17.980628    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50bfa7dd-de16-483f-b499-710061a40640" path="/var/lib/kubelet/pods/50bfa7dd-de16-483f-b499-710061a40640/volumes"
May 25 11:28:18.076342 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.076307    2556 generic.go:358] "Generic (PLEG): container finished" podID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerID="36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062" exitCode=0
May 25 11:28:18.076545 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.076392    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" event={"ID":"128bacb7-b517-4455-b06f-2ac7ee595cf9","Type":"ContainerDied","Data":"36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062"}
May 25 11:28:18.076545 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.076429    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx" event={"ID":"128bacb7-b517-4455-b06f-2ac7ee595cf9","Type":"ContainerDied","Data":"9bf8d40c737d7b2e2a038537bbe37e1127be4cd3a4c970bd026b4e999973e97b"}
May 25 11:28:18.076545 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.076445    2556 scope.go:117] "RemoveContainer" containerID="ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40"
May 25 11:28:18.076545 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.076404    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"
May 25 11:28:18.077925 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.077907    2556 generic.go:358] "Generic (PLEG): container finished" podID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerID="a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f" exitCode=0
May 25 11:28:18.078028 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.077983    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" event={"ID":"de3ccca0-1f20-4bbf-8543-5cb6643d6530","Type":"ContainerDied","Data":"a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f"}
May 25 11:28:18.079605 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.079586    2556 generic.go:358] "Generic (PLEG): container finished" podID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerID="b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0" exitCode=0
May 25 11:28:18.079701 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.079621    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" event={"ID":"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b","Type":"ContainerDied","Data":"b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0"}
May 25 11:28:18.085107 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.085094    2556 scope.go:117] "RemoveContainer" containerID="36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062"
May 25 11:28:18.098163 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.098141    2556 scope.go:117] "RemoveContainer" containerID="e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5"
May 25 11:28:18.117893 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.117875    2556 scope.go:117] "RemoveContainer" containerID="ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40"
May 25 11:28:18.118188 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:28:18.118168    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40\": container with ID starting with ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40 not found: ID does not exist" containerID="ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40"
May 25 11:28:18.118278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.118194    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40"} err="failed to get container status \"ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40\": rpc error: code = NotFound desc = could not find container \"ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40\": container with ID starting with ec70fec82d42df56eb8bef79e60fb40ab4abe69729125e5e5d9acb34dc8f6e40 not found: ID does not exist"
May 25 11:28:18.118278 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.118213    2556 scope.go:117] "RemoveContainer" containerID="36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062"
May 25 11:28:18.118471 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:28:18.118454    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062\": container with ID starting with 36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062 not found: ID does not exist" containerID="36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062"
May 25 11:28:18.118578 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.118474    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062"} err="failed to get container status \"36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062\": rpc error: code = NotFound desc = could not find container \"36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062\": container with ID starting with 36d3b65325411e9d120659392a55dce544ed00ec64abdacf0da2eaebeb825062 not found: ID does not exist"
May 25 11:28:18.118578 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.118506    2556 scope.go:117] "RemoveContainer" containerID="e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5"
May 25 11:28:18.118739 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:28:18.118720    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5\": container with ID starting with e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5 not found: ID does not exist" containerID="e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5"
May 25 11:28:18.118800 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.118743    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5"} err="failed to get container status \"e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5\": rpc error: code = NotFound desc = could not find container \"e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5\": container with ID starting with e94a4dcce8423a549cef6d611ac09fb5a43045a73352d632430bbb71a0eaefb5 not found: ID does not exist"
May 25 11:28:18.137964 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.137940    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"]
May 25 11:28:18.142577 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:18.142553    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-9ad8c-predictor-9885798ff-hfksx"]
May 25 11:28:19.085393 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:19.085354    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" event={"ID":"de3ccca0-1f20-4bbf-8543-5cb6643d6530","Type":"ContainerStarted","Data":"0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489"}
May 25 11:28:19.085393 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:19.085400    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" event={"ID":"de3ccca0-1f20-4bbf-8543-5cb6643d6530","Type":"ContainerStarted","Data":"f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c"}
May 25 11:28:19.085894 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:19.085609    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:19.087030 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:19.087009    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" event={"ID":"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b","Type":"ContainerStarted","Data":"3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9"}
May 25 11:28:19.087030 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:19.087033    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" event={"ID":"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b","Type":"ContainerStarted","Data":"7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b"}
May 25 11:28:19.087249 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:19.087231    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:19.107076 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:19.107028    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podStartSLOduration=7.107014955 podStartE2EDuration="7.107014955s" podCreationTimestamp="2026-05-25 11:28:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:28:19.105986927 +0000 UTC m=+647.736147749" watchObservedRunningTime="2026-05-25 11:28:19.107014955 +0000 UTC m=+647.737175766"
May 25 11:28:19.127631 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:19.127588    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podStartSLOduration=7.127576624 podStartE2EDuration="7.127576624s" podCreationTimestamp="2026-05-25 11:28:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:28:19.125833127 +0000 UTC m=+647.755993937" watchObservedRunningTime="2026-05-25 11:28:19.127576624 +0000 UTC m=+647.757737626"
May 25 11:28:19.980838 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:19.980798    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" path="/var/lib/kubelet/pods/128bacb7-b517-4455-b06f-2ac7ee595cf9/volumes"
May 25 11:28:20.089914 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:20.089884    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:20.090332 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:20.089922    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:20.090814 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:20.090788    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 11:28:20.090895 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:20.090818    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 11:28:21.093001 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:21.092958    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 11:28:21.093355 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:21.092958    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 11:28:26.097995 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:26.097967    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:28:26.100456 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:26.098507    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:28:26.100456 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:26.099107    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 11:28:26.100456 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:26.099441    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 11:28:36.099844 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:36.099806    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 11:28:36.099844 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:36.099824    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 11:28:46.099623 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:46.099586    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 11:28:46.100004 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:46.099586    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 11:28:56.099077 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:56.099035    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 11:28:56.099563 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:28:56.099189    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 11:29:06.099919 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:06.099881    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 11:29:06.100280 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:06.099883    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 11:29:16.099729 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:16.099646    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 11:29:16.100207 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:16.100189    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:29:26.099958 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:26.099928    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:29:53.088978 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.088947    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"]
May 25 11:29:53.089402 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.089252    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" containerID="cri-o://7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b" gracePeriod=30
May 25 11:29:53.089402 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.089295    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kube-rbac-proxy" containerID="cri-o://3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9" gracePeriod=30
May 25 11:29:53.125407 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125379    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"]
May 25 11:29:53.125738 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125726    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="storage-initializer"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125741    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="storage-initializer"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125758    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kube-rbac-proxy"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125763    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kube-rbac-proxy"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125784    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="storage-initializer"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125790    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="storage-initializer"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125797    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125802    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125807    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125812    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125819    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kube-rbac-proxy"
May 25 11:29:53.125825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125824    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kube-rbac-proxy"
May 25 11:29:53.126158 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125885    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kube-rbac-proxy"
May 25 11:29:53.126158 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125893    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="128bacb7-b517-4455-b06f-2ac7ee595cf9" containerName="kserve-container"
May 25 11:29:53.126158 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125903    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kserve-container"
May 25 11:29:53.126158 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.125912    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="50bfa7dd-de16-483f-b499-710061a40640" containerName="kube-rbac-proxy"
May 25 11:29:53.128990 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.128975    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.131308 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.131286    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-56fc4-predictor-serving-cert\""
May 25 11:29:53.131308 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.131301    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\""
May 25 11:29:53.137052 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.137033    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"]
May 25 11:29:53.170189 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.169972    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jclp8\" (UniqueName: \"kubernetes.io/projected/105b482a-e872-4d49-95b9-524c95e0ccc1-kube-api-access-jclp8\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.170189 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.170026    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/105b482a-e872-4d49-95b9-524c95e0ccc1-message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.170189 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.170089    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/105b482a-e872-4d49-95b9-524c95e0ccc1-proxy-tls\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.171775 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.171753    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"]
May 25 11:29:53.172091 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.172068    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container" containerID="cri-o://f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c" gracePeriod=30
May 25 11:29:53.172184 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.172082    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kube-rbac-proxy" containerID="cri-o://0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489" gracePeriod=30
May 25 11:29:53.271301 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.271270    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-jclp8\" (UniqueName: \"kubernetes.io/projected/105b482a-e872-4d49-95b9-524c95e0ccc1-kube-api-access-jclp8\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.271404 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.271314    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/105b482a-e872-4d49-95b9-524c95e0ccc1-message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.271404 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.271366    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/105b482a-e872-4d49-95b9-524c95e0ccc1-proxy-tls\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.271577 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:29:53.271566    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-serving-cert: secret "message-dumper-raw-56fc4-predictor-serving-cert" not found
May 25 11:29:53.271646 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:29:53.271633    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/105b482a-e872-4d49-95b9-524c95e0ccc1-proxy-tls podName:105b482a-e872-4d49-95b9-524c95e0ccc1 nodeName:}" failed. No retries permitted until 2026-05-25 11:29:53.771612048 +0000 UTC m=+742.401772850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/105b482a-e872-4d49-95b9-524c95e0ccc1-proxy-tls") pod "message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" (UID: "105b482a-e872-4d49-95b9-524c95e0ccc1") : secret "message-dumper-raw-56fc4-predictor-serving-cert" not found
May 25 11:29:53.271936 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.271919    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/105b482a-e872-4d49-95b9-524c95e0ccc1-message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.279979 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.279957    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jclp8\" (UniqueName: \"kubernetes.io/projected/105b482a-e872-4d49-95b9-524c95e0ccc1-kube-api-access-jclp8\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.374992 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.374937    2556 generic.go:358] "Generic (PLEG): container finished" podID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerID="0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489" exitCode=2
May 25 11:29:53.375102 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.375020    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" event={"ID":"de3ccca0-1f20-4bbf-8543-5cb6643d6530","Type":"ContainerDied","Data":"0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489"}
May 25 11:29:53.376983 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.376957    2556 generic.go:358] "Generic (PLEG): container finished" podID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerID="3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9" exitCode=2
May 25 11:29:53.377076 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.377006    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" event={"ID":"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b","Type":"ContainerDied","Data":"3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9"}
May 25 11:29:53.775763 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.775728    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/105b482a-e872-4d49-95b9-524c95e0ccc1-proxy-tls\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:53.778209 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:53.778180    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/105b482a-e872-4d49-95b9-524c95e0ccc1-proxy-tls\") pod \"message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") " pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:54.039922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:54.039842    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:54.170292 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:54.170258    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"]
May 25 11:29:54.173367 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:29:54.173333    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod105b482a_e872_4d49_95b9_524c95e0ccc1.slice/crio-c5f0c1925b4158fad6d1b94790880db6674f50375cd68ec96aebfe9d40638064 WatchSource:0}: Error finding container c5f0c1925b4158fad6d1b94790880db6674f50375cd68ec96aebfe9d40638064: Status 404 returned error can't find the container with id c5f0c1925b4158fad6d1b94790880db6674f50375cd68ec96aebfe9d40638064
May 25 11:29:54.175659 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:54.175641    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:29:54.382101 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:54.382011    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" event={"ID":"105b482a-e872-4d49-95b9-524c95e0ccc1","Type":"ContainerStarted","Data":"c5f0c1925b4158fad6d1b94790880db6674f50375cd68ec96aebfe9d40638064"}
May 25 11:29:55.392418 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:55.392391    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" event={"ID":"105b482a-e872-4d49-95b9-524c95e0ccc1","Type":"ContainerStarted","Data":"f80f33d505f4c009b3edeae445f2cf4b26e7e07d04e96642bfd0691d476d375f"}
May 25 11:29:56.094111 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.094076    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
May 25 11:29:56.094300 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.094075    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
May 25 11:29:56.099330 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.099303    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
May 25 11:29:56.099430 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.099306    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
May 25 11:29:56.396812 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.396731    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" event={"ID":"105b482a-e872-4d49-95b9-524c95e0ccc1","Type":"ContainerStarted","Data":"6ceac553dd150f41858f107bce402e8274a77973ae08cd13613ec46b8602dfa3"}
May 25 11:29:56.397144 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.396865    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:56.414665 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.414625    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" podStartSLOduration=2.322683894 podStartE2EDuration="3.414589779s" podCreationTimestamp="2026-05-25 11:29:53 +0000 UTC" firstStartedPulling="2026-05-25 11:29:54.175766778 +0000 UTC m=+742.805927566" lastFinishedPulling="2026-05-25 11:29:55.267672649 +0000 UTC m=+743.897833451" observedRunningTime="2026-05-25 11:29:56.412853623 +0000 UTC m=+745.043014432" watchObservedRunningTime="2026-05-25 11:29:56.414589779 +0000 UTC m=+745.044750589"
May 25 11:29:56.609647 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.609624    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:29:56.701678 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.701647    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3ccca0-1f20-4bbf-8543-5cb6643d6530-isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") pod \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") "
May 25 11:29:56.701810 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.701686    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kserve-provision-location\") pod \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") "
May 25 11:29:56.701810 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.701735    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3ccca0-1f20-4bbf-8543-5cb6643d6530-proxy-tls\") pod \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") "
May 25 11:29:56.701810 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.701773    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48trw\" (UniqueName: \"kubernetes.io/projected/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kube-api-access-48trw\") pod \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\" (UID: \"de3ccca0-1f20-4bbf-8543-5cb6643d6530\") "
May 25 11:29:56.702020 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.701989    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3ccca0-1f20-4bbf-8543-5cb6643d6530-isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config") pod "de3ccca0-1f20-4bbf-8543-5cb6643d6530" (UID: "de3ccca0-1f20-4bbf-8543-5cb6643d6530"). InnerVolumeSpecName "isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:29:56.702108 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.702000    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de3ccca0-1f20-4bbf-8543-5cb6643d6530" (UID: "de3ccca0-1f20-4bbf-8543-5cb6643d6530"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:29:56.703805 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.703784    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de3ccca0-1f20-4bbf-8543-5cb6643d6530-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "de3ccca0-1f20-4bbf-8543-5cb6643d6530" (UID: "de3ccca0-1f20-4bbf-8543-5cb6643d6530"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:29:56.703805 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.703783    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kube-api-access-48trw" (OuterVolumeSpecName: "kube-api-access-48trw") pod "de3ccca0-1f20-4bbf-8543-5cb6643d6530" (UID: "de3ccca0-1f20-4bbf-8543-5cb6643d6530"). InnerVolumeSpecName "kube-api-access-48trw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:29:56.803197 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.803158    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/de3ccca0-1f20-4bbf-8543-5cb6643d6530-isvc-xgboost-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:29:56.803197 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.803193    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:29:56.803396 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.803210    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/de3ccca0-1f20-4bbf-8543-5cb6643d6530-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:29:56.803396 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:56.803225    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48trw\" (UniqueName: \"kubernetes.io/projected/de3ccca0-1f20-4bbf-8543-5cb6643d6530-kube-api-access-48trw\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:29:57.117394 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.117372    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:29:57.206588 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.206548    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") pod \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") "
May 25 11:29:57.206737 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.206607    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kserve-provision-location\") pod \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") "
May 25 11:29:57.206737 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.206654    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-proxy-tls\") pod \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") "
May 25 11:29:57.206737 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.206694    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86xx5\" (UniqueName: \"kubernetes.io/projected/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kube-api-access-86xx5\") pod \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\" (UID: \"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b\") "
May 25 11:29:57.206935 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.206892    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" (UID: "3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:29:57.206935 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.206893    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config") pod "3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" (UID: "3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b"). InnerVolumeSpecName "isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:29:57.208608 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.208588    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kube-api-access-86xx5" (OuterVolumeSpecName: "kube-api-access-86xx5") pod "3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" (UID: "3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b"). InnerVolumeSpecName "kube-api-access-86xx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:29:57.208694 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.208672    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" (UID: "3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:29:57.307641 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.307581    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:29:57.307641 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.307609    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86xx5\" (UniqueName: \"kubernetes.io/projected/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kube-api-access-86xx5\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:29:57.307641 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.307619    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-isvc-sklearn-graph-raw-hpa-6729d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:29:57.307641 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.307629    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:29:57.401006 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.400978    2556 generic.go:358] "Generic (PLEG): container finished" podID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerID="f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c" exitCode=0
May 25 11:29:57.401291 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.401054    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"
May 25 11:29:57.401291 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.401071    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" event={"ID":"de3ccca0-1f20-4bbf-8543-5cb6643d6530","Type":"ContainerDied","Data":"f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c"}
May 25 11:29:57.401291 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.401118    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt" event={"ID":"de3ccca0-1f20-4bbf-8543-5cb6643d6530","Type":"ContainerDied","Data":"28008799e39011b4f0bc2638700308a694eb6e6829bcdf0a07db1d1b3e4f6831"}
May 25 11:29:57.401291 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.401138    2556 scope.go:117] "RemoveContainer" containerID="0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489"
May 25 11:29:57.403065 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.403041    2556 generic.go:358] "Generic (PLEG): container finished" podID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerID="7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b" exitCode=0
May 25 11:29:57.403183 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.403166    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" event={"ID":"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b","Type":"ContainerDied","Data":"7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b"}
May 25 11:29:57.403236 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.403199    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk" event={"ID":"3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b","Type":"ContainerDied","Data":"1d666c082b115c148bf0c2f74b39628794a1c2aecc574c5e19827c9a08273e70"}
May 25 11:29:57.403236 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.403167    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"
May 25 11:29:57.403639 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.403618    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:57.405335 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.405318    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:29:57.409662 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.409646    2556 scope.go:117] "RemoveContainer" containerID="f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c"
May 25 11:29:57.417198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.417183    2556 scope.go:117] "RemoveContainer" containerID="a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f"
May 25 11:29:57.424117 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.424100    2556 scope.go:117] "RemoveContainer" containerID="0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489"
May 25 11:29:57.424348 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:29:57.424330    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489\": container with ID starting with 0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489 not found: ID does not exist" containerID="0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489"
May 25 11:29:57.424386 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.424356    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489"} err="failed to get container status \"0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489\": rpc error: code = NotFound desc = could not find container \"0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489\": container with ID starting with 0bb7bbfa4e970614933a2c69d47cb5dee22bd80deb5cb4ffc888a42fad64a489 not found: ID does not exist"
May 25 11:29:57.424386 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.424375    2556 scope.go:117] "RemoveContainer" containerID="f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c"
May 25 11:29:57.424611 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:29:57.424594    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c\": container with ID starting with f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c not found: ID does not exist" containerID="f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c"
May 25 11:29:57.424653 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.424617    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c"} err="failed to get container status \"f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c\": rpc error: code = NotFound desc = could not find container \"f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c\": container with ID starting with f17bc5de0a20dafae1d5694944a3e115220a18bb10d48178c4e060ab6efbf54c not found: ID does not exist"
May 25 11:29:57.424653 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.424631    2556 scope.go:117] "RemoveContainer" containerID="a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f"
May 25 11:29:57.424835 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:29:57.424821    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f\": container with ID starting with a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f not found: ID does not exist" containerID="a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f"
May 25 11:29:57.424872 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.424838    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f"} err="failed to get container status \"a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f\": rpc error: code = NotFound desc = could not find container \"a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f\": container with ID starting with a135d5179714bdadcdf56981c2db41cc00c58c865c86037ccf96bd81a897ed7f not found: ID does not exist"
May 25 11:29:57.424872 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.424851    2556 scope.go:117] "RemoveContainer" containerID="3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9"
May 25 11:29:57.431894 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.431879    2556 scope.go:117] "RemoveContainer" containerID="7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b"
May 25 11:29:57.438334 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.438278    2556 scope.go:117] "RemoveContainer" containerID="b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0"
May 25 11:29:57.440713 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.440693    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"]
May 25 11:29:57.446268 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.446246    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-6729d-predictor-cc7bbb9f-4kxlk"]
May 25 11:29:57.446595 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.446578    2556 scope.go:117] "RemoveContainer" containerID="3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9"
May 25 11:29:57.446827 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:29:57.446806    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9\": container with ID starting with 3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9 not found: ID does not exist" containerID="3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9"
May 25 11:29:57.446890 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.446834    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9"} err="failed to get container status \"3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9\": rpc error: code = NotFound desc = could not find container \"3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9\": container with ID starting with 3b2e02502c5a3dbaeb73afe9a326bd0bbf8162e71de3fd0220de4e8635cd4ca9 not found: ID does not exist"
May 25 11:29:57.446890 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.446851    2556 scope.go:117] "RemoveContainer" containerID="7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b"
May 25 11:29:57.447092 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:29:57.447078    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b\": container with ID starting with 7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b not found: ID does not exist" containerID="7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b"
May 25 11:29:57.447136 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.447095    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b"} err="failed to get container status \"7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b\": rpc error: code = NotFound desc = could not find container \"7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b\": container with ID starting with 7c385211fa026ec9d9aaba5f15eea5ff1d586d13472a375b2f8df9a84be4742b not found: ID does not exist"
May 25 11:29:57.447136 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.447107    2556 scope.go:117] "RemoveContainer" containerID="b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0"
May 25 11:29:57.447328 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:29:57.447310    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0\": container with ID starting with b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0 not found: ID does not exist" containerID="b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0"
May 25 11:29:57.447376 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.447333    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0"} err="failed to get container status \"b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0\": rpc error: code = NotFound desc = could not find container \"b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0\": container with ID starting with b15cb6b5349e7ba0eb02d234f996ea45edb3297ca1cc007b804fd00619909aa0 not found: ID does not exist"
May 25 11:29:57.457980 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.457960    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"]
May 25 11:29:57.459898 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.459878    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-6729d-predictor-855c97d797-s5zdt"]
May 25 11:29:57.980132 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.980104    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" path="/var/lib/kubelet/pods/3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b/volumes"
May 25 11:29:57.980593 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:29:57.980580    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" path="/var/lib/kubelet/pods/de3ccca0-1f20-4bbf-8543-5cb6643d6530/volumes"
May 25 11:30:04.414686 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:04.414659    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:30:13.233788 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.233754    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"]
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234100    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234111    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234126    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234131    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234136    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kube-rbac-proxy"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234142    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kube-rbac-proxy"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234149    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="storage-initializer"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234154    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="storage-initializer"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234166    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="storage-initializer"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234171    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="storage-initializer"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234179    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kube-rbac-proxy"
May 25 11:30:13.234212 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234184    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kube-rbac-proxy"
May 25 11:30:13.234628 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234250    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kserve-container"
May 25 11:30:13.234628 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234260    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="3ce9ab66-f3ad-4ac9-9d78-29cc7ce36a6b" containerName="kube-rbac-proxy"
May 25 11:30:13.234628 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234268    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kserve-container"
May 25 11:30:13.234628 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.234273    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="de3ccca0-1f20-4bbf-8543-5cb6643d6530" containerName="kube-rbac-proxy"
May 25 11:30:13.242070 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.242041    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.244353 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.244332    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-56fc4-predictor-serving-cert\""
May 25 11:30:13.244614 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.244332    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\""
May 25 11:30:13.247315 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.247285    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"]
May 25 11:30:13.328537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.328509    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f4cc10-07c1-4bec-a6fe-4992459b1060-kserve-provision-location\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.328676 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.328544    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f4cc10-07c1-4bec-a6fe-4992459b1060-isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.328676 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.328624    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccxvl\" (UniqueName: \"kubernetes.io/projected/48f4cc10-07c1-4bec-a6fe-4992459b1060-kube-api-access-ccxvl\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.328766 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.328711    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f4cc10-07c1-4bec-a6fe-4992459b1060-proxy-tls\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.429939 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.429900    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f4cc10-07c1-4bec-a6fe-4992459b1060-kserve-provision-location\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.430080 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.429950    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f4cc10-07c1-4bec-a6fe-4992459b1060-isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.430080 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.430010    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ccxvl\" (UniqueName: \"kubernetes.io/projected/48f4cc10-07c1-4bec-a6fe-4992459b1060-kube-api-access-ccxvl\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.430080 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.430052    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f4cc10-07c1-4bec-a6fe-4992459b1060-proxy-tls\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.430247 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:30:13.430205    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-serving-cert: secret "isvc-logger-raw-56fc4-predictor-serving-cert" not found
May 25 11:30:13.430298 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:30:13.430267    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48f4cc10-07c1-4bec-a6fe-4992459b1060-proxy-tls podName:48f4cc10-07c1-4bec-a6fe-4992459b1060 nodeName:}" failed. No retries permitted until 2026-05-25 11:30:13.930245814 +0000 UTC m=+762.560406603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/48f4cc10-07c1-4bec-a6fe-4992459b1060-proxy-tls") pod "isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" (UID: "48f4cc10-07c1-4bec-a6fe-4992459b1060") : secret "isvc-logger-raw-56fc4-predictor-serving-cert" not found
May 25 11:30:13.430298 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.430276    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f4cc10-07c1-4bec-a6fe-4992459b1060-kserve-provision-location\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.430571 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.430552    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f4cc10-07c1-4bec-a6fe-4992459b1060-isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.440248 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.440227    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccxvl\" (UniqueName: \"kubernetes.io/projected/48f4cc10-07c1-4bec-a6fe-4992459b1060-kube-api-access-ccxvl\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.934680 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.934651    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f4cc10-07c1-4bec-a6fe-4992459b1060-proxy-tls\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:13.936825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:13.936804    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f4cc10-07c1-4bec-a6fe-4992459b1060-proxy-tls\") pod \"isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") " pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:14.153732 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:14.153700    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:14.279507 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:14.279436    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"]
May 25 11:30:14.281672 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:30:14.281637    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48f4cc10_07c1_4bec_a6fe_4992459b1060.slice/crio-1e91245d6ca766e80ab2bca264cce02a75ce430acfdd996bbdbf9df3ec203dc6 WatchSource:0}: Error finding container 1e91245d6ca766e80ab2bca264cce02a75ce430acfdd996bbdbf9df3ec203dc6: Status 404 returned error can't find the container with id 1e91245d6ca766e80ab2bca264cce02a75ce430acfdd996bbdbf9df3ec203dc6
May 25 11:30:14.463388 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:14.463310    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerStarted","Data":"c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9"}
May 25 11:30:14.463388 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:14.463355    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerStarted","Data":"1e91245d6ca766e80ab2bca264cce02a75ce430acfdd996bbdbf9df3ec203dc6"}
May 25 11:30:18.476214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:18.476180    2556 generic.go:358] "Generic (PLEG): container finished" podID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerID="c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9" exitCode=0
May 25 11:30:18.476601 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:18.476251    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerDied","Data":"c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9"}
May 25 11:30:19.481537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:19.481504    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerStarted","Data":"611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f"}
May 25 11:30:19.481884 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:19.481547    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerStarted","Data":"ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388"}
May 25 11:30:19.481884 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:19.481557    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerStarted","Data":"5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51"}
May 25 11:30:19.481884 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:19.481844    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:19.481993 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:19.481977    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:19.483166 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:19.483141    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:30:19.504304 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:19.504264    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podStartSLOduration=6.504251563 podStartE2EDuration="6.504251563s" podCreationTimestamp="2026-05-25 11:30:13 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:30:19.504159573 +0000 UTC m=+768.134320384" watchObservedRunningTime="2026-05-25 11:30:19.504251563 +0000 UTC m=+768.134412374"
May 25 11:30:20.485749 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:20.485714    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:20.486173 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:20.485831    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:30:20.486789 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:20.486762    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:21.489043 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:21.489001    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:30:21.489546 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:21.489522    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:26.494208 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:26.494171    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:30:26.494831 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:26.494795    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:30:26.495139 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:26.495113    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:36.495402 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:36.495355    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:30:36.495864 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:36.495831    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:46.494974 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:46.494885    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:30:46.495413 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:46.495392    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:30:56.495134 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:56.495086    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:30:56.495568 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:30:56.495546    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:31:06.495648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:06.495598    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:31:06.496136 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:06.496111    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:31:16.495238 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:16.495202    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:31:16.495779 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:16.495757    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:31:26.495618 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:26.495588    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:31:26.496238 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:26.495720    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:31:38.260457 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.260422    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq_105b482a-e872-4d49-95b9-524c95e0ccc1/kserve-container/0.log"
May 25 11:31:38.439545 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.439514    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"]
May 25 11:31:38.439938 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.439892    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" containerID="cri-o://5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51" gracePeriod=30
May 25 11:31:38.440066 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.439927    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy" containerID="cri-o://ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388" gracePeriod=30
May 25 11:31:38.440066 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.439927    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" containerID="cri-o://611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f" gracePeriod=30
May 25 11:31:38.490848 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.490817    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"]
May 25 11:31:38.494739 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.494712    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.497117 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.497087    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\""
May 25 11:31:38.497117 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.497100    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-fdff2-predictor-serving-cert\""
May 25 11:31:38.505438 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.505407    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"]
May 25 11:31:38.560295 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.560267    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"]
May 25 11:31:38.560609 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.560558    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" podUID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerName="kserve-container" containerID="cri-o://f80f33d505f4c009b3edeae445f2cf4b26e7e07d04e96642bfd0691d476d375f" gracePeriod=30
May 25 11:31:38.560609 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.560593    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" podUID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerName="kube-rbac-proxy" containerID="cri-o://6ceac553dd150f41858f107bce402e8274a77973ae08cd13613ec46b8602dfa3" gracePeriod=30
May 25 11:31:38.571720 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.571697    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm5rc\" (UniqueName: \"kubernetes.io/projected/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kube-api-access-cm5rc\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.571817 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.571758    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-proxy-tls\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.571817 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.571806    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.571904 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.571861    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.673245 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.673216    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cm5rc\" (UniqueName: \"kubernetes.io/projected/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kube-api-access-cm5rc\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.673367 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.673271    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-proxy-tls\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.673367 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.673311    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.673496 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.673367    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.673811 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.673789    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.674203 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.674183    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.675681 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.675659    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-proxy-tls\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.683604 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.683584    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm5rc\" (UniqueName: \"kubernetes.io/projected/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kube-api-access-cm5rc\") pod \"isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.724441 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.724409    2556 generic.go:358] "Generic (PLEG): container finished" podID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerID="6ceac553dd150f41858f107bce402e8274a77973ae08cd13613ec46b8602dfa3" exitCode=2
May 25 11:31:38.724441 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.724432    2556 generic.go:358] "Generic (PLEG): container finished" podID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerID="f80f33d505f4c009b3edeae445f2cf4b26e7e07d04e96642bfd0691d476d375f" exitCode=2
May 25 11:31:38.724659 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.724492    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" event={"ID":"105b482a-e872-4d49-95b9-524c95e0ccc1","Type":"ContainerDied","Data":"6ceac553dd150f41858f107bce402e8274a77973ae08cd13613ec46b8602dfa3"}
May 25 11:31:38.724659 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.724529    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" event={"ID":"105b482a-e872-4d49-95b9-524c95e0ccc1","Type":"ContainerDied","Data":"f80f33d505f4c009b3edeae445f2cf4b26e7e07d04e96642bfd0691d476d375f"}
May 25 11:31:38.726513 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.726492    2556 generic.go:358] "Generic (PLEG): container finished" podID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerID="ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388" exitCode=2
May 25 11:31:38.726626 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.726519    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerDied","Data":"ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388"}
May 25 11:31:38.793862 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.793842    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:31:38.805677 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.805618    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:38.875190 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.875151    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/105b482a-e872-4d49-95b9-524c95e0ccc1-proxy-tls\") pod \"105b482a-e872-4d49-95b9-524c95e0ccc1\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") "
May 25 11:31:38.875190 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.875191    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jclp8\" (UniqueName: \"kubernetes.io/projected/105b482a-e872-4d49-95b9-524c95e0ccc1-kube-api-access-jclp8\") pod \"105b482a-e872-4d49-95b9-524c95e0ccc1\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") "
May 25 11:31:38.875403 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.875214    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/105b482a-e872-4d49-95b9-524c95e0ccc1-message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\") pod \"105b482a-e872-4d49-95b9-524c95e0ccc1\" (UID: \"105b482a-e872-4d49-95b9-524c95e0ccc1\") "
May 25 11:31:38.875627 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.875596    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105b482a-e872-4d49-95b9-524c95e0ccc1-message-dumper-raw-56fc4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "message-dumper-raw-56fc4-kube-rbac-proxy-sar-config") pod "105b482a-e872-4d49-95b9-524c95e0ccc1" (UID: "105b482a-e872-4d49-95b9-524c95e0ccc1"). InnerVolumeSpecName "message-dumper-raw-56fc4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:31:38.877143 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.877119    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105b482a-e872-4d49-95b9-524c95e0ccc1-kube-api-access-jclp8" (OuterVolumeSpecName: "kube-api-access-jclp8") pod "105b482a-e872-4d49-95b9-524c95e0ccc1" (UID: "105b482a-e872-4d49-95b9-524c95e0ccc1"). InnerVolumeSpecName "kube-api-access-jclp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:31:38.877143 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.877135    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/105b482a-e872-4d49-95b9-524c95e0ccc1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "105b482a-e872-4d49-95b9-524c95e0ccc1" (UID: "105b482a-e872-4d49-95b9-524c95e0ccc1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:31:38.932220 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.932199    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"]
May 25 11:31:38.934754 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:31:38.934729    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ac5e19_5ea5_4c9c_94c3_ba675a552d45.slice/crio-14f412dcf345270b802d4fa1472a5ff03767ba86d1fda01c28ad0f63c9db2057 WatchSource:0}: Error finding container 14f412dcf345270b802d4fa1472a5ff03767ba86d1fda01c28ad0f63c9db2057: Status 404 returned error can't find the container with id 14f412dcf345270b802d4fa1472a5ff03767ba86d1fda01c28ad0f63c9db2057
May 25 11:31:38.976167 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.976148    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/105b482a-e872-4d49-95b9-524c95e0ccc1-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:31:38.976245 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.976171    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jclp8\" (UniqueName: \"kubernetes.io/projected/105b482a-e872-4d49-95b9-524c95e0ccc1-kube-api-access-jclp8\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:31:38.976245 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:38.976182    2556 reconciler_common.go:299] "Volume detached for volume \"message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/105b482a-e872-4d49-95b9-524c95e0ccc1-message-dumper-raw-56fc4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:31:39.731446 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:39.731369    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq" event={"ID":"105b482a-e872-4d49-95b9-524c95e0ccc1","Type":"ContainerDied","Data":"c5f0c1925b4158fad6d1b94790880db6674f50375cd68ec96aebfe9d40638064"}
May 25 11:31:39.731446 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:39.731397    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"
May 25 11:31:39.731446 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:39.731411    2556 scope.go:117] "RemoveContainer" containerID="6ceac553dd150f41858f107bce402e8274a77973ae08cd13613ec46b8602dfa3"
May 25 11:31:39.732767 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:39.732746    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" event={"ID":"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45","Type":"ContainerStarted","Data":"3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e"}
May 25 11:31:39.732831 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:39.732773    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" event={"ID":"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45","Type":"ContainerStarted","Data":"14f412dcf345270b802d4fa1472a5ff03767ba86d1fda01c28ad0f63c9db2057"}
May 25 11:31:39.739725 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:39.739670    2556 scope.go:117] "RemoveContainer" containerID="f80f33d505f4c009b3edeae445f2cf4b26e7e07d04e96642bfd0691d476d375f"
May 25 11:31:39.764927 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:39.764906    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"]
May 25 11:31:39.768818 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:39.768798    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-56fc4-predictor-5d757c866b-5jzjq"]
May 25 11:31:39.981271 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:39.981239    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="105b482a-e872-4d49-95b9-524c95e0ccc1" path="/var/lib/kubelet/pods/105b482a-e872-4d49-95b9-524c95e0ccc1/volumes"
May 25 11:31:41.489812 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:41.489776    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 25 11:31:42.746157 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:42.746123    2556 generic.go:358] "Generic (PLEG): container finished" podID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerID="5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51" exitCode=0
May 25 11:31:42.746520 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:42.746190    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerDied","Data":"5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51"}
May 25 11:31:43.750729 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:43.750697    2556 generic.go:358] "Generic (PLEG): container finished" podID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerID="3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e" exitCode=0
May 25 11:31:43.751070 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:43.750767    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" event={"ID":"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45","Type":"ContainerDied","Data":"3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e"}
May 25 11:31:44.755705 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:44.755673    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" event={"ID":"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45","Type":"ContainerStarted","Data":"1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0"}
May 25 11:31:44.756052 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:44.755716    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" event={"ID":"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45","Type":"ContainerStarted","Data":"0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85"}
May 25 11:31:44.756052 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:44.755911    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:44.778537 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:44.777619    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podStartSLOduration=6.777603281 podStartE2EDuration="6.777603281s" podCreationTimestamp="2026-05-25 11:31:38 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:31:44.775939209 +0000 UTC m=+853.406100019" watchObservedRunningTime="2026-05-25 11:31:44.777603281 +0000 UTC m=+853.407764095"
May 25 11:31:45.759627 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:45.759600    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:45.760620 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:45.760593    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:31:46.489593 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:46.489554    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 25 11:31:46.494919 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:46.494890    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:31:46.495244 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:46.495218    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:31:46.762898 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:46.762823    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:31:51.489241 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:51.489201    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 25 11:31:51.489729 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:51.489349    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:31:51.767394 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:51.767320    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:31:51.767929 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:51.767907    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:31:56.489602 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:56.489566    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 25 11:31:56.495622 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:56.495584    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:31:56.495920 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:31:56.495897    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:32:01.489540 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:01.489477    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 25 11:32:01.768612 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:01.768530    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:32:06.489264 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:06.489225    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
May 25 11:32:06.494723 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:06.494698    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
May 25 11:32:06.494837 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:06.494825    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:32:06.495026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:06.495004    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
May 25 11:32:06.495105 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:06.495094    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:32:08.625277 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.625256    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:32:08.809026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.808958    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f4cc10-07c1-4bec-a6fe-4992459b1060-kserve-provision-location\") pod \"48f4cc10-07c1-4bec-a6fe-4992459b1060\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") "
May 25 11:32:08.809160 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.809061    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f4cc10-07c1-4bec-a6fe-4992459b1060-isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\") pod \"48f4cc10-07c1-4bec-a6fe-4992459b1060\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") "
May 25 11:32:08.809160 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.809101    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f4cc10-07c1-4bec-a6fe-4992459b1060-proxy-tls\") pod \"48f4cc10-07c1-4bec-a6fe-4992459b1060\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") "
May 25 11:32:08.809160 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.809137    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccxvl\" (UniqueName: \"kubernetes.io/projected/48f4cc10-07c1-4bec-a6fe-4992459b1060-kube-api-access-ccxvl\") pod \"48f4cc10-07c1-4bec-a6fe-4992459b1060\" (UID: \"48f4cc10-07c1-4bec-a6fe-4992459b1060\") "
May 25 11:32:08.809363 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.809334    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f4cc10-07c1-4bec-a6fe-4992459b1060-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48f4cc10-07c1-4bec-a6fe-4992459b1060" (UID: "48f4cc10-07c1-4bec-a6fe-4992459b1060"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:32:08.809477 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.809446    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48f4cc10-07c1-4bec-a6fe-4992459b1060-isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config") pod "48f4cc10-07c1-4bec-a6fe-4992459b1060" (UID: "48f4cc10-07c1-4bec-a6fe-4992459b1060"). InnerVolumeSpecName "isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:32:08.811048 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.811024    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f4cc10-07c1-4bec-a6fe-4992459b1060-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "48f4cc10-07c1-4bec-a6fe-4992459b1060" (UID: "48f4cc10-07c1-4bec-a6fe-4992459b1060"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:32:08.811115 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.811083    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f4cc10-07c1-4bec-a6fe-4992459b1060-kube-api-access-ccxvl" (OuterVolumeSpecName: "kube-api-access-ccxvl") pod "48f4cc10-07c1-4bec-a6fe-4992459b1060" (UID: "48f4cc10-07c1-4bec-a6fe-4992459b1060"). InnerVolumeSpecName "kube-api-access-ccxvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:32:08.830404 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.830381    2556 generic.go:358] "Generic (PLEG): container finished" podID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerID="611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f" exitCode=0
May 25 11:32:08.830503 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.830466    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"
May 25 11:32:08.830503 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.830469    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerDied","Data":"611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f"}
May 25 11:32:08.830626 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.830533    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr" event={"ID":"48f4cc10-07c1-4bec-a6fe-4992459b1060","Type":"ContainerDied","Data":"1e91245d6ca766e80ab2bca264cce02a75ce430acfdd996bbdbf9df3ec203dc6"}
May 25 11:32:08.830626 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.830557    2556 scope.go:117] "RemoveContainer" containerID="611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f"
May 25 11:32:08.838355 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.838338    2556 scope.go:117] "RemoveContainer" containerID="ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388"
May 25 11:32:08.845543 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.845526    2556 scope.go:117] "RemoveContainer" containerID="5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51"
May 25 11:32:08.853138 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.853113    2556 scope.go:117] "RemoveContainer" containerID="c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9"
May 25 11:32:08.855002 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.854982    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"]
May 25 11:32:08.860079 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.860060    2556 scope.go:117] "RemoveContainer" containerID="611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f"
May 25 11:32:08.860334 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:32:08.860315    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f\": container with ID starting with 611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f not found: ID does not exist" containerID="611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f"
May 25 11:32:08.860438 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.860345    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f"} err="failed to get container status \"611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f\": rpc error: code = NotFound desc = could not find container \"611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f\": container with ID starting with 611813bcd2e009ca76a8ace56c04aed228038d092ed88f2c6d105a47c7f44c1f not found: ID does not exist"
May 25 11:32:08.860438 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.860364    2556 scope.go:117] "RemoveContainer" containerID="ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388"
May 25 11:32:08.860650 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:32:08.860629    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388\": container with ID starting with ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388 not found: ID does not exist" containerID="ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388"
May 25 11:32:08.860710 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.860659    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388"} err="failed to get container status \"ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388\": rpc error: code = NotFound desc = could not find container \"ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388\": container with ID starting with ef8a8aa5ff71197f732bc8216db41e0b509d7a8e4d9a3d2e541080eec3890388 not found: ID does not exist"
May 25 11:32:08.860710 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.860681    2556 scope.go:117] "RemoveContainer" containerID="5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51"
May 25 11:32:08.861013 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:32:08.860988    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51\": container with ID starting with 5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51 not found: ID does not exist" containerID="5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51"
May 25 11:32:08.861085 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.861020    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51"} err="failed to get container status \"5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51\": rpc error: code = NotFound desc = could not find container \"5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51\": container with ID starting with 5360dc1f3d04d1f3c3308eae82fe53e92be351577cfa6ab9e592e22d86932b51 not found: ID does not exist"
May 25 11:32:08.861085 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.861040    2556 scope.go:117] "RemoveContainer" containerID="c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9"
May 25 11:32:08.861287 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:32:08.861267    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9\": container with ID starting with c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9 not found: ID does not exist" containerID="c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9"
May 25 11:32:08.861372 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.861292    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9"} err="failed to get container status \"c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9\": rpc error: code = NotFound desc = could not find container \"c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9\": container with ID starting with c32c63078a097f153c4686eb9fcccc6c175fd776c6ba76b59d849078b970f8b9 not found: ID does not exist"
May 25 11:32:08.863301 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.863282    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-56fc4-predictor-56fbf8d6c6-g2lhr"]
May 25 11:32:08.910231 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.910211    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48f4cc10-07c1-4bec-a6fe-4992459b1060-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:32:08.910231 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.910231    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/48f4cc10-07c1-4bec-a6fe-4992459b1060-isvc-logger-raw-56fc4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:32:08.910346 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.910243    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/48f4cc10-07c1-4bec-a6fe-4992459b1060-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:32:08.910346 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:08.910253    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccxvl\" (UniqueName: \"kubernetes.io/projected/48f4cc10-07c1-4bec-a6fe-4992459b1060-kube-api-access-ccxvl\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:32:09.979750 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:09.979720    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" path="/var/lib/kubelet/pods/48f4cc10-07c1-4bec-a6fe-4992459b1060/volumes"
May 25 11:32:11.768348 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:11.768308    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:32:21.768571 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:21.768466    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:32:31.768127 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:31.768090    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:32:31.913683 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:31.913653    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:32:31.914701 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:31.914671    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:32:41.768369 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:41.768326    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:32:51.768886 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:32:51.768844    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:33:01.767968 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:01.767931    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:33:10.976509 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:10.976446    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:33:20.976593 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:20.976548    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:33:30.977381 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:30.977333    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:33:40.977159 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:40.977121    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:33:50.976689 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:50.976609    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:33:58.784284 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.784243    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"]
May 25 11:33:58.784870 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.784642    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" containerID="cri-o://0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85" gracePeriod=30
May 25 11:33:58.784870 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.784693    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kube-rbac-proxy" containerID="cri-o://1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0" gracePeriod=30
May 25 11:33:58.940920 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.940896    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"]
May 25 11:33:58.941234 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941222    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy"
May 25 11:33:58.941276 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941236    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy"
May 25 11:33:58.941276 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941246    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container"
May 25 11:33:58.941276 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941252    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container"
May 25 11:33:58.941276 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941261    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="storage-initializer"
May 25 11:33:58.941276 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941267    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="storage-initializer"
May 25 11:33:58.941276 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941275    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerName="kube-rbac-proxy"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941281    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerName="kube-rbac-proxy"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941291    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerName="kserve-container"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941296    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerName="kserve-container"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941305    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941309    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941357    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kube-rbac-proxy"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941365    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="agent"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941373    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="48f4cc10-07c1-4bec-a6fe-4992459b1060" containerName="kserve-container"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941380    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerName="kube-rbac-proxy"
May 25 11:33:58.941443 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.941385    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="105b482a-e872-4d49-95b9-524c95e0ccc1" containerName="kserve-container"
May 25 11:33:58.944370 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.944355    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:58.949107 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.949088    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c840ff-predictor-serving-cert\""
May 25 11:33:58.949496 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.949464    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-c840ff-kube-rbac-proxy-sar-config\""
May 25 11:33:58.967203 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:58.967181    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"]
May 25 11:33:59.032093 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.032069    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ead778-7e6c-4dcb-913d-3365e563bcc5-proxy-tls\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.032221 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.032112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47pk\" (UniqueName: \"kubernetes.io/projected/73ead778-7e6c-4dcb-913d-3365e563bcc5-kube-api-access-p47pk\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.032221 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.032153    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-primary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ead778-7e6c-4dcb-913d-3365e563bcc5-isvc-primary-c840ff-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.032221 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.032172    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ead778-7e6c-4dcb-913d-3365e563bcc5-kserve-provision-location\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.133349 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.133285    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-p47pk\" (UniqueName: \"kubernetes.io/projected/73ead778-7e6c-4dcb-913d-3365e563bcc5-kube-api-access-p47pk\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.133349 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.133327    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-primary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ead778-7e6c-4dcb-913d-3365e563bcc5-isvc-primary-c840ff-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.133349 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.133347    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ead778-7e6c-4dcb-913d-3365e563bcc5-kserve-provision-location\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.133577 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.133403    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ead778-7e6c-4dcb-913d-3365e563bcc5-proxy-tls\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.133577 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:33:59.133539    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-c840ff-predictor-serving-cert: secret "isvc-primary-c840ff-predictor-serving-cert" not found
May 25 11:33:59.133659 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:33:59.133595    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73ead778-7e6c-4dcb-913d-3365e563bcc5-proxy-tls podName:73ead778-7e6c-4dcb-913d-3365e563bcc5 nodeName:}" failed. No retries permitted until 2026-05-25 11:33:59.633574166 +0000 UTC m=+988.263734961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/73ead778-7e6c-4dcb-913d-3365e563bcc5-proxy-tls") pod "isvc-primary-c840ff-predictor-5889595d56-pqb4x" (UID: "73ead778-7e6c-4dcb-913d-3365e563bcc5") : secret "isvc-primary-c840ff-predictor-serving-cert" not found
May 25 11:33:59.133797 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.133777    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ead778-7e6c-4dcb-913d-3365e563bcc5-kserve-provision-location\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.133961 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.133943    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-primary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ead778-7e6c-4dcb-913d-3365e563bcc5-isvc-primary-c840ff-kube-rbac-proxy-sar-config\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.143922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.143900    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47pk\" (UniqueName: \"kubernetes.io/projected/73ead778-7e6c-4dcb-913d-3365e563bcc5-kube-api-access-p47pk\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.150260 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.150232    2556 generic.go:358] "Generic (PLEG): container finished" podID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerID="1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0" exitCode=2
May 25 11:33:59.150362 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.150289    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" event={"ID":"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45","Type":"ContainerDied","Data":"1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0"}
May 25 11:33:59.638219 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.638181    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ead778-7e6c-4dcb-913d-3365e563bcc5-proxy-tls\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.640634 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.640613    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ead778-7e6c-4dcb-913d-3365e563bcc5-proxy-tls\") pod \"isvc-primary-c840ff-predictor-5889595d56-pqb4x\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") " pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.854549 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.854521    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:33:59.992314 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:33:59.992284    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"]
May 25 11:33:59.995465 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:33:59.995439    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ead778_7e6c_4dcb_913d_3365e563bcc5.slice/crio-0dba31e0a74fd66c1096305a0072deb4cd88ff0e06ac830d55e8d7c7e198e888 WatchSource:0}: Error finding container 0dba31e0a74fd66c1096305a0072deb4cd88ff0e06ac830d55e8d7c7e198e888: Status 404 returned error can't find the container with id 0dba31e0a74fd66c1096305a0072deb4cd88ff0e06ac830d55e8d7c7e198e888
May 25 11:34:00.155285 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:00.155215    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" event={"ID":"73ead778-7e6c-4dcb-913d-3365e563bcc5","Type":"ContainerStarted","Data":"432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206"}
May 25 11:34:00.155285 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:00.155249    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" event={"ID":"73ead778-7e6c-4dcb-913d-3365e563bcc5","Type":"ContainerStarted","Data":"0dba31e0a74fd66c1096305a0072deb4cd88ff0e06ac830d55e8d7c7e198e888"}
May 25 11:34:00.976834 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:00.976793    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
May 25 11:34:01.763379 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:01.763338    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
May 25 11:34:04.168370 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:04.168283    2556 generic.go:358] "Generic (PLEG): container finished" podID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerID="432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206" exitCode=0
May 25 11:34:04.168370 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:04.168334    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" event={"ID":"73ead778-7e6c-4dcb-913d-3365e563bcc5","Type":"ContainerDied","Data":"432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206"}
May 25 11:34:05.173465 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:05.173427    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" event={"ID":"73ead778-7e6c-4dcb-913d-3365e563bcc5","Type":"ContainerStarted","Data":"c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe"}
May 25 11:34:05.173465 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:05.173470    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" event={"ID":"73ead778-7e6c-4dcb-913d-3365e563bcc5","Type":"ContainerStarted","Data":"5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451"}
May 25 11:34:05.173908 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:05.173793    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:34:05.173964 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:05.173944    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:34:05.175430 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:05.175401    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 11:34:05.198192 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:05.198150    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podStartSLOduration=7.198137833 podStartE2EDuration="7.198137833s" podCreationTimestamp="2026-05-25 11:33:58 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:34:05.196263622 +0000 UTC m=+993.826424432" watchObservedRunningTime="2026-05-25 11:34:05.198137833 +0000 UTC m=+993.828298643"
May 25 11:34:06.176560 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:06.176514    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 11:34:06.763676 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:06.763635    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
May 25 11:34:07.025309 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.025247    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:34:07.099666 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.099645    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm5rc\" (UniqueName: \"kubernetes.io/projected/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kube-api-access-cm5rc\") pod \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") "
May 25 11:34:07.099792 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.099740    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-proxy-tls\") pod \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") "
May 25 11:34:07.099792 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.099759    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kserve-provision-location\") pod \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") "
May 25 11:34:07.099866 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.099793    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\") pod \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\" (UID: \"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45\") "
May 25 11:34:07.100258 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.100104    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" (UID: "f0ac5e19-5ea5-4c9c-94c3-ba675a552d45"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:34:07.100258 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.100196    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config") pod "f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" (UID: "f0ac5e19-5ea5-4c9c-94c3-ba675a552d45"). InnerVolumeSpecName "isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:34:07.101684 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.101663    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kube-api-access-cm5rc" (OuterVolumeSpecName: "kube-api-access-cm5rc") pod "f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" (UID: "f0ac5e19-5ea5-4c9c-94c3-ba675a552d45"). InnerVolumeSpecName "kube-api-access-cm5rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:34:07.101775 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.101755    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" (UID: "f0ac5e19-5ea5-4c9c-94c3-ba675a552d45"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:34:07.181052 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.181027    2556 generic.go:358] "Generic (PLEG): container finished" podID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerID="0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85" exitCode=0
May 25 11:34:07.181413 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.181119    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"
May 25 11:34:07.181413 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.181114    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" event={"ID":"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45","Type":"ContainerDied","Data":"0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85"}
May 25 11:34:07.181413 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.181231    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj" event={"ID":"f0ac5e19-5ea5-4c9c-94c3-ba675a552d45","Type":"ContainerDied","Data":"14f412dcf345270b802d4fa1472a5ff03767ba86d1fda01c28ad0f63c9db2057"}
May 25 11:34:07.181413 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.181246    2556 scope.go:117] "RemoveContainer" containerID="1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0"
May 25 11:34:07.189261 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.189242    2556 scope.go:117] "RemoveContainer" containerID="0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85"
May 25 11:34:07.196118 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.196104    2556 scope.go:117] "RemoveContainer" containerID="3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e"
May 25 11:34:07.201223 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.201206    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:34:07.201305 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.201224    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:34:07.201305 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.201235    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-isvc-sklearn-scale-raw-fdff2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:34:07.201305 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.201246    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm5rc\" (UniqueName: \"kubernetes.io/projected/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45-kube-api-access-cm5rc\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:34:07.203198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.203182    2556 scope.go:117] "RemoveContainer" containerID="1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0"
May 25 11:34:07.203453 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:34:07.203431    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0\": container with ID starting with 1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0 not found: ID does not exist" containerID="1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0"
May 25 11:34:07.203538 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.203464    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0"} err="failed to get container status \"1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0\": rpc error: code = NotFound desc = could not find container \"1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0\": container with ID starting with 1c39245d5428f88c895fa58cec404b1d3d5d5577b5b4eb78d7a25bede5dc9ed0 not found: ID does not exist"
May 25 11:34:07.203538 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.203507    2556 scope.go:117] "RemoveContainer" containerID="0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85"
May 25 11:34:07.203775 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:34:07.203757    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85\": container with ID starting with 0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85 not found: ID does not exist" containerID="0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85"
May 25 11:34:07.203775 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.203782    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85"} err="failed to get container status \"0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85\": rpc error: code = NotFound desc = could not find container \"0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85\": container with ID starting with 0c15e487c6895119d364c223876d10e03ee9250aa6855f22864a6a5af2223f85 not found: ID does not exist"
May 25 11:34:07.203775 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.203800    2556 scope.go:117] "RemoveContainer" containerID="3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e"
May 25 11:34:07.204027 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:34:07.204007    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e\": container with ID starting with 3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e not found: ID does not exist" containerID="3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e"
May 25 11:34:07.204088 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.204033    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e"} err="failed to get container status \"3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e\": rpc error: code = NotFound desc = could not find container \"3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e\": container with ID starting with 3dc7d79e91df7c21c9bada60fca9fe2da5f798c017d6759707e3c0051f97a26e not found: ID does not exist"
May 25 11:34:07.204543 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.204524    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"]
May 25 11:34:07.207637 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.207613    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-fdff2-predictor-58654984bf-cn7bj"]
May 25 11:34:07.982197 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:07.982161    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" path="/var/lib/kubelet/pods/f0ac5e19-5ea5-4c9c-94c3-ba675a552d45/volumes"
May 25 11:34:11.180606 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:11.180580    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:34:11.181081 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:11.181056    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 11:34:21.181071 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:21.181028    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 11:34:31.181188 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:31.181139    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 11:34:41.180999 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:41.180960    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 11:34:51.181405 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:34:51.181363    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 11:35:01.182021 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:01.181985    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
May 25 11:35:11.182058 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:11.182029    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:35:19.038328 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.038257    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"]
May 25 11:35:19.038765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.038616    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container"
May 25 11:35:19.038765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.038627    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container"
May 25 11:35:19.038765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.038636    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="storage-initializer"
May 25 11:35:19.038765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.038641    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="storage-initializer"
May 25 11:35:19.038765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.038650    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kube-rbac-proxy"
May 25 11:35:19.038765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.038655    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kube-rbac-proxy"
May 25 11:35:19.038765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.038710    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kube-rbac-proxy"
May 25 11:35:19.038765 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.038718    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="f0ac5e19-5ea5-4c9c-94c3-ba675a552d45" containerName="kserve-container"
May 25 11:35:19.041693 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.041677    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.045379 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.045353    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c840ff-kube-rbac-proxy-sar-config\""
May 25 11:35:19.045379 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.045364    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-c840ff\""
May 25 11:35:19.045379 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.045378    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-c840ff-predictor-serving-cert\""
May 25 11:35:19.045379 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.045378    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-c840ff-dockercfg-gpl4g\""
May 25 11:35:19.045642 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.045419    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
May 25 11:35:19.053106 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.053085    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"]
May 25 11:35:19.144994 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.144969    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-cabundle-cert\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.145134 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.145022    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-secondary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-isvc-secondary-c840ff-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.145134 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.145114    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcs5w\" (UniqueName: \"kubernetes.io/projected/546dd5ae-b885-4c62-9797-09b11a39ea6c-kube-api-access-bcs5w\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.145211 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.145152    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/546dd5ae-b885-4c62-9797-09b11a39ea6c-proxy-tls\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.145211 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.145174    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/546dd5ae-b885-4c62-9797-09b11a39ea6c-kserve-provision-location\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.246082 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.246058    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-cabundle-cert\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.246206 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.246092    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-secondary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-isvc-secondary-c840ff-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.246206 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.246130    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bcs5w\" (UniqueName: \"kubernetes.io/projected/546dd5ae-b885-4c62-9797-09b11a39ea6c-kube-api-access-bcs5w\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.246206 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.246155    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/546dd5ae-b885-4c62-9797-09b11a39ea6c-proxy-tls\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.246206 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.246181    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/546dd5ae-b885-4c62-9797-09b11a39ea6c-kserve-provision-location\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.246651 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.246621    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/546dd5ae-b885-4c62-9797-09b11a39ea6c-kserve-provision-location\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.246788 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.246728    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-secondary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-isvc-secondary-c840ff-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.246848 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.246817    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-cabundle-cert\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.248465 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.248443    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/546dd5ae-b885-4c62-9797-09b11a39ea6c-proxy-tls\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.256088 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.256068    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcs5w\" (UniqueName: \"kubernetes.io/projected/546dd5ae-b885-4c62-9797-09b11a39ea6c-kube-api-access-bcs5w\") pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") " pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.352448 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.352401    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:19.483469 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.483436    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"]
May 25 11:35:19.485704 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:35:19.485646    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546dd5ae_b885_4c62_9797_09b11a39ea6c.slice/crio-28e39d243f05bc1f82c39c49faa18cdf9bb52c44abacfe2f0967996861f7f515 WatchSource:0}: Error finding container 28e39d243f05bc1f82c39c49faa18cdf9bb52c44abacfe2f0967996861f7f515: Status 404 returned error can't find the container with id 28e39d243f05bc1f82c39c49faa18cdf9bb52c44abacfe2f0967996861f7f515
May 25 11:35:19.487636 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:19.487618    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
May 25 11:35:20.385555 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:20.385514    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6" event={"ID":"546dd5ae-b885-4c62-9797-09b11a39ea6c","Type":"ContainerStarted","Data":"ca6bef16831ffa5ab878d09c795d3b92c6565c2d45336c57520412b595d7c27e"}
May 25 11:35:20.385555 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:20.385556    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6" event={"ID":"546dd5ae-b885-4c62-9797-09b11a39ea6c","Type":"ContainerStarted","Data":"28e39d243f05bc1f82c39c49faa18cdf9bb52c44abacfe2f0967996861f7f515"}
May 25 11:35:22.392690 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:22.392665    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_546dd5ae-b885-4c62-9797-09b11a39ea6c/storage-initializer/0.log"
May 25 11:35:22.392977 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:22.392708    2556 generic.go:358] "Generic (PLEG): container finished" podID="546dd5ae-b885-4c62-9797-09b11a39ea6c" containerID="ca6bef16831ffa5ab878d09c795d3b92c6565c2d45336c57520412b595d7c27e" exitCode=1
May 25 11:35:22.392977 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:22.392786    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6" event={"ID":"546dd5ae-b885-4c62-9797-09b11a39ea6c","Type":"ContainerDied","Data":"ca6bef16831ffa5ab878d09c795d3b92c6565c2d45336c57520412b595d7c27e"}
May 25 11:35:23.397429 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:23.397401    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_546dd5ae-b885-4c62-9797-09b11a39ea6c/storage-initializer/0.log"
May 25 11:35:23.397814 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:23.397461    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6" event={"ID":"546dd5ae-b885-4c62-9797-09b11a39ea6c","Type":"ContainerStarted","Data":"cb420518cd76488eefcbeabf1f5a7476495831edde4a7b5aaa74eb8051ad06f2"}
May 25 11:35:26.412931 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:26.412899    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_546dd5ae-b885-4c62-9797-09b11a39ea6c/storage-initializer/1.log"
May 25 11:35:26.413313 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:26.413274    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_546dd5ae-b885-4c62-9797-09b11a39ea6c/storage-initializer/0.log"
May 25 11:35:26.413365 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:26.413308    2556 generic.go:358] "Generic (PLEG): container finished" podID="546dd5ae-b885-4c62-9797-09b11a39ea6c" containerID="cb420518cd76488eefcbeabf1f5a7476495831edde4a7b5aaa74eb8051ad06f2" exitCode=1
May 25 11:35:26.413413 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:26.413388    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6" event={"ID":"546dd5ae-b885-4c62-9797-09b11a39ea6c","Type":"ContainerDied","Data":"cb420518cd76488eefcbeabf1f5a7476495831edde4a7b5aaa74eb8051ad06f2"}
May 25 11:35:26.413453 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:26.413440    2556 scope.go:117] "RemoveContainer" containerID="ca6bef16831ffa5ab878d09c795d3b92c6565c2d45336c57520412b595d7c27e"
May 25 11:35:26.413811 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:26.413797    2556 scope.go:117] "RemoveContainer" containerID="ca6bef16831ffa5ab878d09c795d3b92c6565c2d45336c57520412b595d7c27e"
May 25 11:35:26.425977 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:26.425953    2556 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_kserve-ci-e2e-test_546dd5ae-b885-4c62-9797-09b11a39ea6c_0 in pod sandbox 28e39d243f05bc1f82c39c49faa18cdf9bb52c44abacfe2f0967996861f7f515 from index: no such id: 'ca6bef16831ffa5ab878d09c795d3b92c6565c2d45336c57520412b595d7c27e'" containerID="ca6bef16831ffa5ab878d09c795d3b92c6565c2d45336c57520412b595d7c27e"
May 25 11:35:26.426041 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:26.425995    2556 kuberuntime_container.go:951] "Unhandled Error" err="failed to remove pod init container \"storage-initializer\": rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_kserve-ci-e2e-test_546dd5ae-b885-4c62-9797-09b11a39ea6c_0 in pod sandbox 28e39d243f05bc1f82c39c49faa18cdf9bb52c44abacfe2f0967996861f7f515 from index: no such id: 'ca6bef16831ffa5ab878d09c795d3b92c6565c2d45336c57520412b595d7c27e'; Skipping pod \"isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_kserve-ci-e2e-test(546dd5ae-b885-4c62-9797-09b11a39ea6c)\"" logger="UnhandledError"
May 25 11:35:26.427436 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:26.427416    2556 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_kserve-ci-e2e-test(546dd5ae-b885-4c62-9797-09b11a39ea6c)\"" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6" podUID="546dd5ae-b885-4c62-9797-09b11a39ea6c"
May 25 11:35:27.418240 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:27.418216    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_546dd5ae-b885-4c62-9797-09b11a39ea6c/storage-initializer/1.log"
May 25 11:35:33.242115 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.242083    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"]
May 25 11:35:33.242591 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.242385    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container" containerID="cri-o://5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451" gracePeriod=30
May 25 11:35:33.242591 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.242444    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kube-rbac-proxy" containerID="cri-o://c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe" gracePeriod=30
May 25 11:35:33.387478 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.387450    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"]
May 25 11:35:33.438448 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.438414    2556 generic.go:358] "Generic (PLEG): container finished" podID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerID="c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe" exitCode=2
May 25 11:35:33.438647 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.438576    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" event={"ID":"73ead778-7e6c-4dcb-913d-3365e563bcc5","Type":"ContainerDied","Data":"c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe"}
May 25 11:35:33.519183 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.519158    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_546dd5ae-b885-4c62-9797-09b11a39ea6c/storage-initializer/1.log"
May 25 11:35:33.519277 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.519220    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:33.579280 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.579255    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"]
May 25 11:35:33.579685 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.579671    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546dd5ae-b885-4c62-9797-09b11a39ea6c" containerName="storage-initializer"
May 25 11:35:33.579747 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.579687    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="546dd5ae-b885-4c62-9797-09b11a39ea6c" containerName="storage-initializer"
May 25 11:35:33.579787 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.579754    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="546dd5ae-b885-4c62-9797-09b11a39ea6c" containerName="storage-initializer"
May 25 11:35:33.579851 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.579842    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546dd5ae-b885-4c62-9797-09b11a39ea6c" containerName="storage-initializer"
May 25 11:35:33.579884 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.579851    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="546dd5ae-b885-4c62-9797-09b11a39ea6c" containerName="storage-initializer"
May 25 11:35:33.579917 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.579901    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="546dd5ae-b885-4c62-9797-09b11a39ea6c" containerName="storage-initializer"
May 25 11:35:33.583026 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.583010    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.587456 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.587424    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\""
May 25 11:35:33.587591 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.587535    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-e289d9\""
May 25 11:35:33.587787 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.587769    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-e289d9-dockercfg-s6zvq\""
May 25 11:35:33.587860 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.587849    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-e289d9-predictor-serving-cert\""
May 25 11:35:33.619431 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.619408    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"]
May 25 11:35:33.660429 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.660407    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-cabundle-cert\") pod \"546dd5ae-b885-4c62-9797-09b11a39ea6c\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") "
May 25 11:35:33.660589 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.660446    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/546dd5ae-b885-4c62-9797-09b11a39ea6c-proxy-tls\") pod \"546dd5ae-b885-4c62-9797-09b11a39ea6c\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") "
May 25 11:35:33.660589 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.660527    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcs5w\" (UniqueName: \"kubernetes.io/projected/546dd5ae-b885-4c62-9797-09b11a39ea6c-kube-api-access-bcs5w\") pod \"546dd5ae-b885-4c62-9797-09b11a39ea6c\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") "
May 25 11:35:33.660589 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.660562    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/546dd5ae-b885-4c62-9797-09b11a39ea6c-kserve-provision-location\") pod \"546dd5ae-b885-4c62-9797-09b11a39ea6c\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") "
May 25 11:35:33.660771 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.660619    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-secondary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-isvc-secondary-c840ff-kube-rbac-proxy-sar-config\") pod \"546dd5ae-b885-4c62-9797-09b11a39ea6c\" (UID: \"546dd5ae-b885-4c62-9797-09b11a39ea6c\") "
May 25 11:35:33.660869 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.660831    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.660869 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.660843    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/546dd5ae-b885-4c62-9797-09b11a39ea6c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "546dd5ae-b885-4c62-9797-09b11a39ea6c" (UID: "546dd5ae-b885-4c62-9797-09b11a39ea6c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:33.661045 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.660895    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "546dd5ae-b885-4c62-9797-09b11a39ea6c" (UID: "546dd5ae-b885-4c62-9797-09b11a39ea6c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:35:33.661045 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.660967    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58s9j\" (UniqueName: \"kubernetes.io/projected/d01d1ea1-c06b-4bae-82b4-e1084d570162-kube-api-access-58s9j\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.661045 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.661040    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d01d1ea1-c06b-4bae-82b4-e1084d570162-proxy-tls\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.661178 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.661091    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d01d1ea1-c06b-4bae-82b4-e1084d570162-kserve-provision-location\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.661178 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.661149    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-cabundle-cert\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.661178 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.661154    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-isvc-secondary-c840ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-secondary-c840ff-kube-rbac-proxy-sar-config") pod "546dd5ae-b885-4c62-9797-09b11a39ea6c" (UID: "546dd5ae-b885-4c62-9797-09b11a39ea6c"). InnerVolumeSpecName "isvc-secondary-c840ff-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:35:33.661283 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.661237    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-cabundle-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:33.661283 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.661252    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/546dd5ae-b885-4c62-9797-09b11a39ea6c-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:33.661283 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.661262    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-secondary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/546dd5ae-b885-4c62-9797-09b11a39ea6c-isvc-secondary-c840ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:33.662632 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.662604    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546dd5ae-b885-4c62-9797-09b11a39ea6c-kube-api-access-bcs5w" (OuterVolumeSpecName: "kube-api-access-bcs5w") pod "546dd5ae-b885-4c62-9797-09b11a39ea6c" (UID: "546dd5ae-b885-4c62-9797-09b11a39ea6c"). InnerVolumeSpecName "kube-api-access-bcs5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:35:33.662743 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.662718    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/546dd5ae-b885-4c62-9797-09b11a39ea6c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "546dd5ae-b885-4c62-9797-09b11a39ea6c" (UID: "546dd5ae-b885-4c62-9797-09b11a39ea6c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:35:33.761694 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.761629    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.761694 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.761682    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-58s9j\" (UniqueName: \"kubernetes.io/projected/d01d1ea1-c06b-4bae-82b4-e1084d570162-kube-api-access-58s9j\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.761858 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.761705    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d01d1ea1-c06b-4bae-82b4-e1084d570162-proxy-tls\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.761858 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.761722    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d01d1ea1-c06b-4bae-82b4-e1084d570162-kserve-provision-location\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.761858 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.761747    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-cabundle-cert\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.761858 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.761780    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/546dd5ae-b885-4c62-9797-09b11a39ea6c-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:33.761858 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.761795    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcs5w\" (UniqueName: \"kubernetes.io/projected/546dd5ae-b885-4c62-9797-09b11a39ea6c-kube-api-access-bcs5w\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:33.761858 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:33.761852    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-serving-cert: secret "isvc-init-fail-e289d9-predictor-serving-cert" not found
May 25 11:35:33.762096 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:33.761923    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d01d1ea1-c06b-4bae-82b4-e1084d570162-proxy-tls podName:d01d1ea1-c06b-4bae-82b4-e1084d570162 nodeName:}" failed. No retries permitted until 2026-05-25 11:35:34.261898977 +0000 UTC m=+1082.892059767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d01d1ea1-c06b-4bae-82b4-e1084d570162-proxy-tls") pod "isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn" (UID: "d01d1ea1-c06b-4bae-82b4-e1084d570162") : secret "isvc-init-fail-e289d9-predictor-serving-cert" not found
May 25 11:35:33.762167 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.762095    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d01d1ea1-c06b-4bae-82b4-e1084d570162-kserve-provision-location\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.762352 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.762331    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.762560 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.762541    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-cabundle-cert\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:33.781374 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:33.781350    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58s9j\" (UniqueName: \"kubernetes.io/projected/d01d1ea1-c06b-4bae-82b4-e1084d570162-kube-api-access-58s9j\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:34.265208 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.265182    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d01d1ea1-c06b-4bae-82b4-e1084d570162-proxy-tls\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:34.267528 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.267502    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d01d1ea1-c06b-4bae-82b4-e1084d570162-proxy-tls\") pod \"isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") " pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:34.442792 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.442771    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6_546dd5ae-b885-4c62-9797-09b11a39ea6c/storage-initializer/1.log"
May 25 11:35:34.442920 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.442857    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6" event={"ID":"546dd5ae-b885-4c62-9797-09b11a39ea6c","Type":"ContainerDied","Data":"28e39d243f05bc1f82c39c49faa18cdf9bb52c44abacfe2f0967996861f7f515"}
May 25 11:35:34.442920 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.442878    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"
May 25 11:35:34.442920 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.442886    2556 scope.go:117] "RemoveContainer" containerID="cb420518cd76488eefcbeabf1f5a7476495831edde4a7b5aaa74eb8051ad06f2"
May 25 11:35:34.492651 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.492629    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:34.494242 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.494219    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"]
May 25 11:35:34.498442 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.498419    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-c840ff-predictor-576b7bd8cf-vrml6"]
May 25 11:35:34.647737 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:34.647708    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"]
May 25 11:35:34.650623 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:35:34.650594    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01d1ea1_c06b_4bae_82b4_e1084d570162.slice/crio-5aa8cc35325bae8253e1da07ed379f4632d9752467912496f34ac46591302490 WatchSource:0}: Error finding container 5aa8cc35325bae8253e1da07ed379f4632d9752467912496f34ac46591302490: Status 404 returned error can't find the container with id 5aa8cc35325bae8253e1da07ed379f4632d9752467912496f34ac46591302490
May 25 11:35:35.447284 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:35.447244    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn" event={"ID":"d01d1ea1-c06b-4bae-82b4-e1084d570162","Type":"ContainerStarted","Data":"a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461"}
May 25 11:35:35.447727 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:35.447291    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn" event={"ID":"d01d1ea1-c06b-4bae-82b4-e1084d570162","Type":"ContainerStarted","Data":"5aa8cc35325bae8253e1da07ed379f4632d9752467912496f34ac46591302490"}
May 25 11:35:35.980794 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:35.980761    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="546dd5ae-b885-4c62-9797-09b11a39ea6c" path="/var/lib/kubelet/pods/546dd5ae-b885-4c62-9797-09b11a39ea6c/volumes"
May 25 11:35:36.177302 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:36.177268    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
May 25 11:35:37.089980 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.089959    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:35:37.192991 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.192966    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-primary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ead778-7e6c-4dcb-913d-3365e563bcc5-isvc-primary-c840ff-kube-rbac-proxy-sar-config\") pod \"73ead778-7e6c-4dcb-913d-3365e563bcc5\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") "
May 25 11:35:37.193110 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.193017    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ead778-7e6c-4dcb-913d-3365e563bcc5-proxy-tls\") pod \"73ead778-7e6c-4dcb-913d-3365e563bcc5\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") "
May 25 11:35:37.193152 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.193129    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ead778-7e6c-4dcb-913d-3365e563bcc5-kserve-provision-location\") pod \"73ead778-7e6c-4dcb-913d-3365e563bcc5\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") "
May 25 11:35:37.193200 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.193189    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p47pk\" (UniqueName: \"kubernetes.io/projected/73ead778-7e6c-4dcb-913d-3365e563bcc5-kube-api-access-p47pk\") pod \"73ead778-7e6c-4dcb-913d-3365e563bcc5\" (UID: \"73ead778-7e6c-4dcb-913d-3365e563bcc5\") "
May 25 11:35:37.193413 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.193389    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ead778-7e6c-4dcb-913d-3365e563bcc5-isvc-primary-c840ff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-primary-c840ff-kube-rbac-proxy-sar-config") pod "73ead778-7e6c-4dcb-913d-3365e563bcc5" (UID: "73ead778-7e6c-4dcb-913d-3365e563bcc5"). InnerVolumeSpecName "isvc-primary-c840ff-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:35:37.193472 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.193411    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73ead778-7e6c-4dcb-913d-3365e563bcc5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "73ead778-7e6c-4dcb-913d-3365e563bcc5" (UID: "73ead778-7e6c-4dcb-913d-3365e563bcc5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:37.195000 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.194973    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ead778-7e6c-4dcb-913d-3365e563bcc5-kube-api-access-p47pk" (OuterVolumeSpecName: "kube-api-access-p47pk") pod "73ead778-7e6c-4dcb-913d-3365e563bcc5" (UID: "73ead778-7e6c-4dcb-913d-3365e563bcc5"). InnerVolumeSpecName "kube-api-access-p47pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:35:37.195091 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.195036    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73ead778-7e6c-4dcb-913d-3365e563bcc5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "73ead778-7e6c-4dcb-913d-3365e563bcc5" (UID: "73ead778-7e6c-4dcb-913d-3365e563bcc5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:35:37.294717 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.294685    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/73ead778-7e6c-4dcb-913d-3365e563bcc5-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:37.294717 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.294717    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p47pk\" (UniqueName: \"kubernetes.io/projected/73ead778-7e6c-4dcb-913d-3365e563bcc5-kube-api-access-p47pk\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:37.294850 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.294729    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-primary-c840ff-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/73ead778-7e6c-4dcb-913d-3365e563bcc5-isvc-primary-c840ff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:37.294850 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.294740    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/73ead778-7e6c-4dcb-913d-3365e563bcc5-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:37.455588 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.455534    2556 generic.go:358] "Generic (PLEG): container finished" podID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerID="5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451" exitCode=0
May 25 11:35:37.455662 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.455611    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"
May 25 11:35:37.455662 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.455619    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" event={"ID":"73ead778-7e6c-4dcb-913d-3365e563bcc5","Type":"ContainerDied","Data":"5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451"}
May 25 11:35:37.455662 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.455657    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x" event={"ID":"73ead778-7e6c-4dcb-913d-3365e563bcc5","Type":"ContainerDied","Data":"0dba31e0a74fd66c1096305a0072deb4cd88ff0e06ac830d55e8d7c7e198e888"}
May 25 11:35:37.455890 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.455672    2556 scope.go:117] "RemoveContainer" containerID="c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe"
May 25 11:35:37.464305 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.464290    2556 scope.go:117] "RemoveContainer" containerID="5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451"
May 25 11:35:37.473440 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.473423    2556 scope.go:117] "RemoveContainer" containerID="432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206"
May 25 11:35:37.479964 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.479947    2556 scope.go:117] "RemoveContainer" containerID="c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe"
May 25 11:35:37.480206 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:37.480186    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe\": container with ID starting with c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe not found: ID does not exist" containerID="c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe"
May 25 11:35:37.480277 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.480216    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe"} err="failed to get container status \"c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe\": rpc error: code = NotFound desc = could not find container \"c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe\": container with ID starting with c95c115b5780966f712285e0ffd8a1976bd9de426d892f5a209f8d208a8039fe not found: ID does not exist"
May 25 11:35:37.480277 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.480239    2556 scope.go:117] "RemoveContainer" containerID="5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451"
May 25 11:35:37.480460 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:37.480442    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451\": container with ID starting with 5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451 not found: ID does not exist" containerID="5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451"
May 25 11:35:37.480550 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.480468    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451"} err="failed to get container status \"5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451\": rpc error: code = NotFound desc = could not find container \"5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451\": container with ID starting with 5224b662a1b1b4b44e6b93c7f1fb5222e144f0377187eb202849548504ba6451 not found: ID does not exist"
May 25 11:35:37.480550 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.480530    2556 scope.go:117] "RemoveContainer" containerID="432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206"
May 25 11:35:37.480752 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:37.480735    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206\": container with ID starting with 432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206 not found: ID does not exist" containerID="432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206"
May 25 11:35:37.480804 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.480759    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206"} err="failed to get container status \"432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206\": rpc error: code = NotFound desc = could not find container \"432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206\": container with ID starting with 432ab1c58fcaa2dd97c8bdface4a7f32df3dd813c68895719f6bc1babe302206 not found: ID does not exist"
May 25 11:35:37.488881 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.488853    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"]
May 25 11:35:37.492307 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.492288    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-c840ff-predictor-5889595d56-pqb4x"]
May 25 11:35:37.980040 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:37.980004    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" path="/var/lib/kubelet/pods/73ead778-7e6c-4dcb-913d-3365e563bcc5/volumes"
May 25 11:35:39.464015 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:39.463988    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn_d01d1ea1-c06b-4bae-82b4-e1084d570162/storage-initializer/0.log"
May 25 11:35:39.464451 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:39.464028    2556 generic.go:358] "Generic (PLEG): container finished" podID="d01d1ea1-c06b-4bae-82b4-e1084d570162" containerID="a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461" exitCode=1
May 25 11:35:39.464451 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:39.464085    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn" event={"ID":"d01d1ea1-c06b-4bae-82b4-e1084d570162","Type":"ContainerDied","Data":"a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461"}
May 25 11:35:40.468803 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:40.468777    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn_d01d1ea1-c06b-4bae-82b4-e1084d570162/storage-initializer/0.log"
May 25 11:35:40.469176 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:40.468830    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn" event={"ID":"d01d1ea1-c06b-4bae-82b4-e1084d570162","Type":"ContainerStarted","Data":"6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87"}
May 25 11:35:43.440769 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.440735    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"]
May 25 11:35:43.441191 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.440994    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn" podUID="d01d1ea1-c06b-4bae-82b4-e1084d570162" containerName="storage-initializer" containerID="cri-o://6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87" gracePeriod=30
May 25 11:35:43.652618 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.652587    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"]
May 25 11:35:43.652941 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.652929    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container"
May 25 11:35:43.652941 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.652942    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container"
May 25 11:35:43.653013 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.652958    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kube-rbac-proxy"
May 25 11:35:43.653013 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.652964    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kube-rbac-proxy"
May 25 11:35:43.653013 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.652974    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="storage-initializer"
May 25 11:35:43.653013 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.652981    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="storage-initializer"
May 25 11:35:43.653150 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.653027    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kube-rbac-proxy"
May 25 11:35:43.653150 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.653037    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="73ead778-7e6c-4dcb-913d-3365e563bcc5" containerName="kserve-container"
May 25 11:35:43.656431 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.656413    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.659772 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.659753    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-e5e4b-predictor-serving-cert\""
May 25 11:35:43.661734 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.661706    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\""
May 25 11:35:43.661869 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.661846    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qhlr\""
May 25 11:35:43.686194 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.686167    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"]
May 25 11:35:43.749201 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.749145    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0746d0-bc8a-402b-8463-c4f82de0f333-raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.749297 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.749210    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdks2\" (UniqueName: \"kubernetes.io/projected/4b0746d0-bc8a-402b-8463-c4f82de0f333-kube-api-access-kdks2\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.749297 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.749234    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0746d0-bc8a-402b-8463-c4f82de0f333-proxy-tls\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.749297 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.749257    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0746d0-bc8a-402b-8463-c4f82de0f333-kserve-provision-location\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.850103 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.850074    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kdks2\" (UniqueName: \"kubernetes.io/projected/4b0746d0-bc8a-402b-8463-c4f82de0f333-kube-api-access-kdks2\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.850233 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.850113    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0746d0-bc8a-402b-8463-c4f82de0f333-proxy-tls\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.850233 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.850138    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0746d0-bc8a-402b-8463-c4f82de0f333-kserve-provision-location\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.850233 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.850170    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0746d0-bc8a-402b-8463-c4f82de0f333-raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.850398 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:43.850250    2556 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-serving-cert: secret "raw-sklearn-e5e4b-predictor-serving-cert" not found
May 25 11:35:43.850398 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:43.850326    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b0746d0-bc8a-402b-8463-c4f82de0f333-proxy-tls podName:4b0746d0-bc8a-402b-8463-c4f82de0f333 nodeName:}" failed. No retries permitted until 2026-05-25 11:35:44.350306083 +0000 UTC m=+1092.980466879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4b0746d0-bc8a-402b-8463-c4f82de0f333-proxy-tls") pod "raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" (UID: "4b0746d0-bc8a-402b-8463-c4f82de0f333") : secret "raw-sklearn-e5e4b-predictor-serving-cert" not found
May 25 11:35:43.850589 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.850567    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0746d0-bc8a-402b-8463-c4f82de0f333-kserve-provision-location\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.850825 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.850806    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0746d0-bc8a-402b-8463-c4f82de0f333-raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.875978 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.875955    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdks2\" (UniqueName: \"kubernetes.io/projected/4b0746d0-bc8a-402b-8463-c4f82de0f333-kube-api-access-kdks2\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:43.981555 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.981534    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn_d01d1ea1-c06b-4bae-82b4-e1084d570162/storage-initializer/1.log"
May 25 11:35:43.981861 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.981844    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn_d01d1ea1-c06b-4bae-82b4-e1084d570162/storage-initializer/0.log"
May 25 11:35:43.981918 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:43.981907    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:44.152343 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.152275    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58s9j\" (UniqueName: \"kubernetes.io/projected/d01d1ea1-c06b-4bae-82b4-e1084d570162-kube-api-access-58s9j\") pod \"d01d1ea1-c06b-4bae-82b4-e1084d570162\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") "
May 25 11:35:44.152343 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.152311    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d01d1ea1-c06b-4bae-82b4-e1084d570162-proxy-tls\") pod \"d01d1ea1-c06b-4bae-82b4-e1084d570162\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") "
May 25 11:35:44.152550 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.152350    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d01d1ea1-c06b-4bae-82b4-e1084d570162-kserve-provision-location\") pod \"d01d1ea1-c06b-4bae-82b4-e1084d570162\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") "
May 25 11:35:44.152550 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.152373    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\") pod \"d01d1ea1-c06b-4bae-82b4-e1084d570162\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") "
May 25 11:35:44.152550 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.152392    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-cabundle-cert\") pod \"d01d1ea1-c06b-4bae-82b4-e1084d570162\" (UID: \"d01d1ea1-c06b-4bae-82b4-e1084d570162\") "
May 25 11:35:44.152706 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.152671    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01d1ea1-c06b-4bae-82b4-e1084d570162-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d01d1ea1-c06b-4bae-82b4-e1084d570162" (UID: "d01d1ea1-c06b-4bae-82b4-e1084d570162"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:35:44.152799 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.152779    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-isvc-init-fail-e289d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "isvc-init-fail-e289d9-kube-rbac-proxy-sar-config") pod "d01d1ea1-c06b-4bae-82b4-e1084d570162" (UID: "d01d1ea1-c06b-4bae-82b4-e1084d570162"). InnerVolumeSpecName "isvc-init-fail-e289d9-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:35:44.152855 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.152816    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d01d1ea1-c06b-4bae-82b4-e1084d570162" (UID: "d01d1ea1-c06b-4bae-82b4-e1084d570162"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:35:44.154374 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.154354    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01d1ea1-c06b-4bae-82b4-e1084d570162-kube-api-access-58s9j" (OuterVolumeSpecName: "kube-api-access-58s9j") pod "d01d1ea1-c06b-4bae-82b4-e1084d570162" (UID: "d01d1ea1-c06b-4bae-82b4-e1084d570162"). InnerVolumeSpecName "kube-api-access-58s9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:35:44.154430 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.154354    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01d1ea1-c06b-4bae-82b4-e1084d570162-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d01d1ea1-c06b-4bae-82b4-e1084d570162" (UID: "d01d1ea1-c06b-4bae-82b4-e1084d570162"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:35:44.253433 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.253399    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58s9j\" (UniqueName: \"kubernetes.io/projected/d01d1ea1-c06b-4bae-82b4-e1084d570162-kube-api-access-58s9j\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:44.253433 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.253429    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d01d1ea1-c06b-4bae-82b4-e1084d570162-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:44.253596 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.253444    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d01d1ea1-c06b-4bae-82b4-e1084d570162-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:44.253596 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.253458    2556 reconciler_common.go:299] "Volume detached for volume \"isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-isvc-init-fail-e289d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:44.253596 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.253472    2556 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d01d1ea1-c06b-4bae-82b4-e1084d570162-cabundle-cert\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:35:44.354604 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.354570    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0746d0-bc8a-402b-8463-c4f82de0f333-proxy-tls\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:44.356724 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.356706    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0746d0-bc8a-402b-8463-c4f82de0f333-proxy-tls\") pod \"raw-sklearn-e5e4b-predictor-5655bd6787-vzg77\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") " pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:44.482293 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.482272    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn_d01d1ea1-c06b-4bae-82b4-e1084d570162/storage-initializer/1.log"
May 25 11:35:44.482757 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.482643    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn_d01d1ea1-c06b-4bae-82b4-e1084d570162/storage-initializer/0.log"
May 25 11:35:44.482757 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.482686    2556 generic.go:358] "Generic (PLEG): container finished" podID="d01d1ea1-c06b-4bae-82b4-e1084d570162" containerID="6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87" exitCode=1
May 25 11:35:44.482838 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.482760    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"
May 25 11:35:44.482838 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.482768    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn" event={"ID":"d01d1ea1-c06b-4bae-82b4-e1084d570162","Type":"ContainerDied","Data":"6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87"}
May 25 11:35:44.482838 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.482803    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn" event={"ID":"d01d1ea1-c06b-4bae-82b4-e1084d570162","Type":"ContainerDied","Data":"5aa8cc35325bae8253e1da07ed379f4632d9752467912496f34ac46591302490"}
May 25 11:35:44.482838 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.482821    2556 scope.go:117] "RemoveContainer" containerID="6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87"
May 25 11:35:44.490890 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.490875    2556 scope.go:117] "RemoveContainer" containerID="a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461"
May 25 11:35:44.497632 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.497616    2556 scope.go:117] "RemoveContainer" containerID="6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87"
May 25 11:35:44.497869 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:44.497851    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87\": container with ID starting with 6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87 not found: ID does not exist" containerID="6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87"
May 25 11:35:44.497922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.497877    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87"} err="failed to get container status \"6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87\": rpc error: code = NotFound desc = could not find container \"6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87\": container with ID starting with 6d1417cec6a8e85f94f62a455276fb4534786c4f0de3ecb67145fc3a327d1d87 not found: ID does not exist"
May 25 11:35:44.497922 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.497894    2556 scope.go:117] "RemoveContainer" containerID="a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461"
May 25 11:35:44.498122 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:35:44.498106    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461\": container with ID starting with a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461 not found: ID does not exist" containerID="a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461"
May 25 11:35:44.498171 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.498132    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461"} err="failed to get container status \"a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461\": rpc error: code = NotFound desc = could not find container \"a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461\": container with ID starting with a922cde88a2df671c1701a6a5c4046e1dd7ab24e3634789a2956a1ab4eac1461 not found: ID does not exist"
May 25 11:35:44.536038 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.536008    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"]
May 25 11:35:44.544382 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.544361    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-e289d9-predictor-bfc49b68f-24ppn"]
May 25 11:35:44.566129 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.566105    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:44.701010 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:44.700984    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"]
May 25 11:35:44.704173 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:35:44.704150    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b0746d0_bc8a_402b_8463_c4f82de0f333.slice/crio-6061bf57d689cd39276577ab45d73a85bde913faf2e6a2d9e62bd61eba83088b WatchSource:0}: Error finding container 6061bf57d689cd39276577ab45d73a85bde913faf2e6a2d9e62bd61eba83088b: Status 404 returned error can't find the container with id 6061bf57d689cd39276577ab45d73a85bde913faf2e6a2d9e62bd61eba83088b
May 25 11:35:45.488270 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:45.488238    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" event={"ID":"4b0746d0-bc8a-402b-8463-c4f82de0f333","Type":"ContainerStarted","Data":"60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d"}
May 25 11:35:45.488270 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:45.488274    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" event={"ID":"4b0746d0-bc8a-402b-8463-c4f82de0f333","Type":"ContainerStarted","Data":"6061bf57d689cd39276577ab45d73a85bde913faf2e6a2d9e62bd61eba83088b"}
May 25 11:35:45.980323 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:45.980294    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d01d1ea1-c06b-4bae-82b4-e1084d570162" path="/var/lib/kubelet/pods/d01d1ea1-c06b-4bae-82b4-e1084d570162/volumes"
May 25 11:35:49.501610 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:49.501574    2556 generic.go:358] "Generic (PLEG): container finished" podID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerID="60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d" exitCode=0
May 25 11:35:49.501959 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:49.501647    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" event={"ID":"4b0746d0-bc8a-402b-8463-c4f82de0f333","Type":"ContainerDied","Data":"60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d"}
May 25 11:35:50.506457 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:50.506425    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" event={"ID":"4b0746d0-bc8a-402b-8463-c4f82de0f333","Type":"ContainerStarted","Data":"0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2"}
May 25 11:35:50.507022 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:50.506469    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" event={"ID":"4b0746d0-bc8a-402b-8463-c4f82de0f333","Type":"ContainerStarted","Data":"59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80"}
May 25 11:35:50.507022 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:50.506730    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:51.509944 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:51.509909    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:51.511112 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:51.511087    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 11:35:52.512500 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:52.512447    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 11:35:57.516632 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:57.516603    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:35:57.517226 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:57.517202    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 11:35:57.565704 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:35:57.565606    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podStartSLOduration=14.565591067 podStartE2EDuration="14.565591067s" podCreationTimestamp="2026-05-25 11:35:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:35:50.546018543 +0000 UTC m=+1099.176179353" watchObservedRunningTime="2026-05-25 11:35:57.565591067 +0000 UTC m=+1106.195751878"
May 25 11:36:07.517787 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:36:07.517747    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 11:36:17.517935 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:36:17.517898    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 11:36:27.517621 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:36:27.517580    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 11:36:37.517820 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:36:37.517771    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 11:36:47.517898 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:36:47.517815    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
May 25 11:36:57.517671 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:36:57.517642    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:37:03.601530 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.601493    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"]
May 25 11:37:03.601974 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.601820    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container" containerID="cri-o://59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80" gracePeriod=30
May 25 11:37:03.601974 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.601848    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kube-rbac-proxy" containerID="cri-o://0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2" gracePeriod=30
May 25 11:37:03.734285 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.734232    2556 generic.go:358] "Generic (PLEG): container finished" podID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerID="0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2" exitCode=2
May 25 11:37:03.734417 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.734309    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" event={"ID":"4b0746d0-bc8a-402b-8463-c4f82de0f333","Type":"ContainerDied","Data":"0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2"}
May 25 11:37:03.740406 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.740378    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"]
May 25 11:37:03.740766 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.740753    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d01d1ea1-c06b-4bae-82b4-e1084d570162" containerName="storage-initializer"
May 25 11:37:03.740814 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.740768    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d01d1ea1-c06b-4bae-82b4-e1084d570162" containerName="storage-initializer"
May 25 11:37:03.740853 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.740831    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="d01d1ea1-c06b-4bae-82b4-e1084d570162" containerName="storage-initializer"
May 25 11:37:03.740899 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.740890    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d01d1ea1-c06b-4bae-82b4-e1084d570162" containerName="storage-initializer"
May 25 11:37:03.740933 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.740898    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d01d1ea1-c06b-4bae-82b4-e1084d570162" containerName="storage-initializer"
May 25 11:37:03.740966 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.740952    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="d01d1ea1-c06b-4bae-82b4-e1084d570162" containerName="storage-initializer"
May 25 11:37:03.744072 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.744053    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.747092 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.747073    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\""
May 25 11:37:03.747219 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.747108    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-1bc1a-predictor-serving-cert\""
May 25 11:37:03.757358 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.757332    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"]
May 25 11:37:03.855127 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.855049    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/552589d3-45c7-4814-82c4-2a4d8e7d5718-raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.855127 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.855087    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/552589d3-45c7-4814-82c4-2a4d8e7d5718-kserve-provision-location\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.855299 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.855185    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/552589d3-45c7-4814-82c4-2a4d8e7d5718-proxy-tls\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.855299 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.855222    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tppr\" (UniqueName: \"kubernetes.io/projected/552589d3-45c7-4814-82c4-2a4d8e7d5718-kube-api-access-6tppr\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.956646 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.956608    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/552589d3-45c7-4814-82c4-2a4d8e7d5718-proxy-tls\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.956833 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.956657    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6tppr\" (UniqueName: \"kubernetes.io/projected/552589d3-45c7-4814-82c4-2a4d8e7d5718-kube-api-access-6tppr\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.956833 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.956712    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/552589d3-45c7-4814-82c4-2a4d8e7d5718-raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.956833 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.956747    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/552589d3-45c7-4814-82c4-2a4d8e7d5718-kserve-provision-location\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.957168 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.957146    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/552589d3-45c7-4814-82c4-2a4d8e7d5718-kserve-provision-location\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.957338 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.957318    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/552589d3-45c7-4814-82c4-2a4d8e7d5718-raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.959030 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.959011    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/552589d3-45c7-4814-82c4-2a4d8e7d5718-proxy-tls\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:03.964989 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:03.964962    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tppr\" (UniqueName: \"kubernetes.io/projected/552589d3-45c7-4814-82c4-2a4d8e7d5718-kube-api-access-6tppr\") pod \"raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:04.054401 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:04.054355    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:04.181214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:04.181181    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"]
May 25 11:37:04.738414 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:04.738376    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" event={"ID":"552589d3-45c7-4814-82c4-2a4d8e7d5718","Type":"ContainerStarted","Data":"02d10bac5a5b466e930a3d4d5a6e6588dc4c24d90a03c78ee24269f5c1e64582"}
May 25 11:37:04.738414 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:04.738415    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" event={"ID":"552589d3-45c7-4814-82c4-2a4d8e7d5718","Type":"ContainerStarted","Data":"f60cc303d446c5a91ad5bf113a3a2a2b0455e0943e24ed49f46f81c7d8bfff03"}
May 25 11:37:07.441879 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.441857    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:37:07.588749 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.588684    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0746d0-bc8a-402b-8463-c4f82de0f333-raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\") pod \"4b0746d0-bc8a-402b-8463-c4f82de0f333\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") "
May 25 11:37:07.588749 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.588718    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0746d0-bc8a-402b-8463-c4f82de0f333-proxy-tls\") pod \"4b0746d0-bc8a-402b-8463-c4f82de0f333\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") "
May 25 11:37:07.588900 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.588760    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0746d0-bc8a-402b-8463-c4f82de0f333-kserve-provision-location\") pod \"4b0746d0-bc8a-402b-8463-c4f82de0f333\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") "
May 25 11:37:07.588900 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.588789    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdks2\" (UniqueName: \"kubernetes.io/projected/4b0746d0-bc8a-402b-8463-c4f82de0f333-kube-api-access-kdks2\") pod \"4b0746d0-bc8a-402b-8463-c4f82de0f333\" (UID: \"4b0746d0-bc8a-402b-8463-c4f82de0f333\") "
May 25 11:37:07.589101 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.589074    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b0746d0-bc8a-402b-8463-c4f82de0f333-raw-sklearn-e5e4b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-e5e4b-kube-rbac-proxy-sar-config") pod "4b0746d0-bc8a-402b-8463-c4f82de0f333" (UID: "4b0746d0-bc8a-402b-8463-c4f82de0f333"). InnerVolumeSpecName "raw-sklearn-e5e4b-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:37:07.589139 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.589091    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b0746d0-bc8a-402b-8463-c4f82de0f333-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b0746d0-bc8a-402b-8463-c4f82de0f333" (UID: "4b0746d0-bc8a-402b-8463-c4f82de0f333"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:37:07.590836 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.590817    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b0746d0-bc8a-402b-8463-c4f82de0f333-kube-api-access-kdks2" (OuterVolumeSpecName: "kube-api-access-kdks2") pod "4b0746d0-bc8a-402b-8463-c4f82de0f333" (UID: "4b0746d0-bc8a-402b-8463-c4f82de0f333"). InnerVolumeSpecName "kube-api-access-kdks2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:37:07.590951 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.590933    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b0746d0-bc8a-402b-8463-c4f82de0f333-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b0746d0-bc8a-402b-8463-c4f82de0f333" (UID: "4b0746d0-bc8a-402b-8463-c4f82de0f333"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:37:07.689754 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.689728    2556 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b0746d0-bc8a-402b-8463-c4f82de0f333-raw-sklearn-e5e4b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:37:07.689754 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.689752    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b0746d0-bc8a-402b-8463-c4f82de0f333-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:37:07.689906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.689762    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b0746d0-bc8a-402b-8463-c4f82de0f333-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:37:07.689906 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.689771    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdks2\" (UniqueName: \"kubernetes.io/projected/4b0746d0-bc8a-402b-8463-c4f82de0f333-kube-api-access-kdks2\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:37:07.749040 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.749008    2556 generic.go:358] "Generic (PLEG): container finished" podID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerID="59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80" exitCode=0
May 25 11:37:07.749184 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.749054    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" event={"ID":"4b0746d0-bc8a-402b-8463-c4f82de0f333","Type":"ContainerDied","Data":"59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80"}
May 25 11:37:07.749184 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.749100    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77" event={"ID":"4b0746d0-bc8a-402b-8463-c4f82de0f333","Type":"ContainerDied","Data":"6061bf57d689cd39276577ab45d73a85bde913faf2e6a2d9e62bd61eba83088b"}
May 25 11:37:07.749184 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.749104    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"
May 25 11:37:07.749184 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.749121    2556 scope.go:117] "RemoveContainer" containerID="0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2"
May 25 11:37:07.758260 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.758233    2556 scope.go:117] "RemoveContainer" containerID="59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80"
May 25 11:37:07.781447 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.781404    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"]
May 25 11:37:07.794976 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.794955    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-e5e4b-predictor-5655bd6787-vzg77"]
May 25 11:37:07.809849 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.809823    2556 scope.go:117] "RemoveContainer" containerID="60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d"
May 25 11:37:07.816423 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.816407    2556 scope.go:117] "RemoveContainer" containerID="0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2"
May 25 11:37:07.816654 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:37:07.816638    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2\": container with ID starting with 0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2 not found: ID does not exist" containerID="0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2"
May 25 11:37:07.816709 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.816662    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2"} err="failed to get container status \"0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2\": rpc error: code = NotFound desc = could not find container \"0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2\": container with ID starting with 0a47a9800ab988171e11f86809527a4959ed90b63dc17d15c601e341a21a01b2 not found: ID does not exist"
May 25 11:37:07.816709 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.816683    2556 scope.go:117] "RemoveContainer" containerID="59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80"
May 25 11:37:07.816928 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:37:07.816911    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80\": container with ID starting with 59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80 not found: ID does not exist" containerID="59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80"
May 25 11:37:07.816976 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.816938    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80"} err="failed to get container status \"59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80\": rpc error: code = NotFound desc = could not find container \"59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80\": container with ID starting with 59a6e183a6623f8ca877f56d7146ec26c8f4ca013d970a74cce7a1c95a30ba80 not found: ID does not exist"
May 25 11:37:07.816976 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.816955    2556 scope.go:117] "RemoveContainer" containerID="60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d"
May 25 11:37:07.817137 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:37:07.817123    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d\": container with ID starting with 60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d not found: ID does not exist" containerID="60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d"
May 25 11:37:07.817173 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.817141    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d"} err="failed to get container status \"60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d\": rpc error: code = NotFound desc = could not find container \"60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d\": container with ID starting with 60b89da9e9d0e601f679efa5fb282b2a9fa868eb8bc4e8298b4308cdd53ef44d not found: ID does not exist"
May 25 11:37:07.980085 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:07.980060    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" path="/var/lib/kubelet/pods/4b0746d0-bc8a-402b-8463-c4f82de0f333/volumes"
May 25 11:37:08.753369 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:08.753337    2556 generic.go:358] "Generic (PLEG): container finished" podID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerID="02d10bac5a5b466e930a3d4d5a6e6588dc4c24d90a03c78ee24269f5c1e64582" exitCode=0
May 25 11:37:08.753816 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:08.753414    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" event={"ID":"552589d3-45c7-4814-82c4-2a4d8e7d5718","Type":"ContainerDied","Data":"02d10bac5a5b466e930a3d4d5a6e6588dc4c24d90a03c78ee24269f5c1e64582"}
May 25 11:37:09.761297 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:09.761265    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" event={"ID":"552589d3-45c7-4814-82c4-2a4d8e7d5718","Type":"ContainerStarted","Data":"3ab8db91d8989512f18aea62927b88e4ce745546c701f5955f5a2f461d613479"}
May 25 11:37:09.761701 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:09.761305    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" event={"ID":"552589d3-45c7-4814-82c4-2a4d8e7d5718","Type":"ContainerStarted","Data":"d545d4ef2346d7a8392d161b541efd923fc0f131b1b54c9eccaf1324e7be3268"}
May 25 11:37:09.761701 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:09.761512    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:09.780317 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:09.780258    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podStartSLOduration=6.780241515 podStartE2EDuration="6.780241515s" podCreationTimestamp="2026-05-25 11:37:03 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:37:09.778870572 +0000 UTC m=+1178.409031382" watchObservedRunningTime="2026-05-25 11:37:09.780241515 +0000 UTC m=+1178.410402326"
May 25 11:37:10.764942 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:10.764900    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:10.766624 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:10.766591    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 11:37:11.768327 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:11.768282    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 11:37:16.772718 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:16.772685    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:37:16.773330 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:16.773305    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 11:37:26.773302 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:26.773264    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 11:37:31.938191 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:31.938149    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:37:31.939897 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:31.939874    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:37:36.774198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:36.774144    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 11:37:46.773699 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:46.773651    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 11:37:56.773214 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:37:56.773174    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 11:38:06.773198 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:06.773161    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 11:38:16.779576 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:16.779504    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:38:23.787267 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:23.787233    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"]
May 25 11:38:23.787751 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:23.787679    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" containerID="cri-o://d545d4ef2346d7a8392d161b541efd923fc0f131b1b54c9eccaf1324e7be3268" gracePeriod=30
May 25 11:38:23.787831 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:23.787712    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kube-rbac-proxy" containerID="cri-o://3ab8db91d8989512f18aea62927b88e4ce745546c701f5955f5a2f461d613479" gracePeriod=30
May 25 11:38:23.982657 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:23.982631    2556 generic.go:358] "Generic (PLEG): container finished" podID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerID="3ab8db91d8989512f18aea62927b88e4ce745546c701f5955f5a2f461d613479" exitCode=2
May 25 11:38:23.982763 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:23.982699    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" event={"ID":"552589d3-45c7-4814-82c4-2a4d8e7d5718","Type":"ContainerDied","Data":"3ab8db91d8989512f18aea62927b88e4ce745546c701f5955f5a2f461d613479"}
May 25 11:38:25.174948 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.174912    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5fplw/must-gather-8mbz5"]
May 25 11:38:25.175429 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.175410    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kube-rbac-proxy"
May 25 11:38:25.175529 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.175432    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kube-rbac-proxy"
May 25 11:38:25.175529 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.175449    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="storage-initializer"
May 25 11:38:25.175529 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.175458    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="storage-initializer"
May 25 11:38:25.175529 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.175496    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container"
May 25 11:38:25.175529 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.175506    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container"
May 25 11:38:25.175782 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.175604    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kube-rbac-proxy"
May 25 11:38:25.175782 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.175618    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="4b0746d0-bc8a-402b-8463-c4f82de0f333" containerName="kserve-container"
May 25 11:38:25.178684 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.178663    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:25.181219 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.181198    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5fplw\"/\"openshift-service-ca.crt\""
May 25 11:38:25.181326 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.181197    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5fplw\"/\"kube-root-ca.crt\""
May 25 11:38:25.182048 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.182032    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5fplw\"/\"default-dockercfg-k276g\""
May 25 11:38:25.185785 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.185764    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5fplw/must-gather-8mbz5"]
May 25 11:38:25.272013 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.271993    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5448ccbb-988b-49a5-8a85-980ad8e428a0-must-gather-output\") pod \"must-gather-8mbz5\" (UID: \"5448ccbb-988b-49a5-8a85-980ad8e428a0\") " pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:25.272127 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.272057    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8kd\" (UniqueName: \"kubernetes.io/projected/5448ccbb-988b-49a5-8a85-980ad8e428a0-kube-api-access-5t8kd\") pod \"must-gather-8mbz5\" (UID: \"5448ccbb-988b-49a5-8a85-980ad8e428a0\") " pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:25.373390 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.373357    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8kd\" (UniqueName: \"kubernetes.io/projected/5448ccbb-988b-49a5-8a85-980ad8e428a0-kube-api-access-5t8kd\") pod \"must-gather-8mbz5\" (UID: \"5448ccbb-988b-49a5-8a85-980ad8e428a0\") " pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:25.373556 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.373419    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5448ccbb-988b-49a5-8a85-980ad8e428a0-must-gather-output\") pod \"must-gather-8mbz5\" (UID: \"5448ccbb-988b-49a5-8a85-980ad8e428a0\") " pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:25.373740 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.373723    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5448ccbb-988b-49a5-8a85-980ad8e428a0-must-gather-output\") pod \"must-gather-8mbz5\" (UID: \"5448ccbb-988b-49a5-8a85-980ad8e428a0\") " pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:25.382093 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.382074    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8kd\" (UniqueName: \"kubernetes.io/projected/5448ccbb-988b-49a5-8a85-980ad8e428a0-kube-api-access-5t8kd\") pod \"must-gather-8mbz5\" (UID: \"5448ccbb-988b-49a5-8a85-980ad8e428a0\") " pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:25.494458 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.494405    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:25.618181 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.618147    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5fplw/must-gather-8mbz5"]
May 25 11:38:25.621453 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:38:25.621419    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5448ccbb_988b_49a5_8a85_980ad8e428a0.slice/crio-ab2ba83aac197dc9b217840ec3b07cf0c878d3741cf58944fbc318e01f1ab8ca WatchSource:0}: Error finding container ab2ba83aac197dc9b217840ec3b07cf0c878d3741cf58944fbc318e01f1ab8ca: Status 404 returned error can't find the container with id ab2ba83aac197dc9b217840ec3b07cf0c878d3741cf58944fbc318e01f1ab8ca
May 25 11:38:25.988248 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:25.988220    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5fplw/must-gather-8mbz5" event={"ID":"5448ccbb-988b-49a5-8a85-980ad8e428a0","Type":"ContainerStarted","Data":"ab2ba83aac197dc9b217840ec3b07cf0c878d3741cf58944fbc318e01f1ab8ca"}
May 25 11:38:26.769159 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:26.769100    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
May 25 11:38:26.773649 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:26.773581    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
May 25 11:38:30.004102 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:30.004068    2556 generic.go:358] "Generic (PLEG): container finished" podID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerID="d545d4ef2346d7a8392d161b541efd923fc0f131b1b54c9eccaf1324e7be3268" exitCode=0
May 25 11:38:30.004522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:30.004196    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" event={"ID":"552589d3-45c7-4814-82c4-2a4d8e7d5718","Type":"ContainerDied","Data":"d545d4ef2346d7a8392d161b541efd923fc0f131b1b54c9eccaf1324e7be3268"}
May 25 11:38:31.615495 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.615459    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:38:31.627267 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.627242    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tppr\" (UniqueName: \"kubernetes.io/projected/552589d3-45c7-4814-82c4-2a4d8e7d5718-kube-api-access-6tppr\") pod \"552589d3-45c7-4814-82c4-2a4d8e7d5718\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") "
May 25 11:38:31.627390 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.627281    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/552589d3-45c7-4814-82c4-2a4d8e7d5718-raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\") pod \"552589d3-45c7-4814-82c4-2a4d8e7d5718\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") "
May 25 11:38:31.627390 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.627312    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/552589d3-45c7-4814-82c4-2a4d8e7d5718-kserve-provision-location\") pod \"552589d3-45c7-4814-82c4-2a4d8e7d5718\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") "
May 25 11:38:31.627390 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.627359    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/552589d3-45c7-4814-82c4-2a4d8e7d5718-proxy-tls\") pod \"552589d3-45c7-4814-82c4-2a4d8e7d5718\" (UID: \"552589d3-45c7-4814-82c4-2a4d8e7d5718\") "
May 25 11:38:31.627737 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.627683    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552589d3-45c7-4814-82c4-2a4d8e7d5718-raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config") pod "552589d3-45c7-4814-82c4-2a4d8e7d5718" (UID: "552589d3-45c7-4814-82c4-2a4d8e7d5718"). InnerVolumeSpecName "raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
May 25 11:38:31.627737 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.627695    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/552589d3-45c7-4814-82c4-2a4d8e7d5718-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "552589d3-45c7-4814-82c4-2a4d8e7d5718" (UID: "552589d3-45c7-4814-82c4-2a4d8e7d5718"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:38:31.629335 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.629307    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552589d3-45c7-4814-82c4-2a4d8e7d5718-kube-api-access-6tppr" (OuterVolumeSpecName: "kube-api-access-6tppr") pod "552589d3-45c7-4814-82c4-2a4d8e7d5718" (UID: "552589d3-45c7-4814-82c4-2a4d8e7d5718"). InnerVolumeSpecName "kube-api-access-6tppr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:38:31.629651 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.629628    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/552589d3-45c7-4814-82c4-2a4d8e7d5718-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "552589d3-45c7-4814-82c4-2a4d8e7d5718" (UID: "552589d3-45c7-4814-82c4-2a4d8e7d5718"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
May 25 11:38:31.728771 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.728752    2556 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/552589d3-45c7-4814-82c4-2a4d8e7d5718-kserve-provision-location\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:38:31.728835 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.728775    2556 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/552589d3-45c7-4814-82c4-2a4d8e7d5718-proxy-tls\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:38:31.728835 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.728785    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tppr\" (UniqueName: \"kubernetes.io/projected/552589d3-45c7-4814-82c4-2a4d8e7d5718-kube-api-access-6tppr\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:38:31.728835 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:31.728795    2556 reconciler_common.go:299] "Volume detached for volume \"raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/552589d3-45c7-4814-82c4-2a4d8e7d5718-raw-sklearn-runtime-1bc1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:38:32.015618 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.015589    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc" event={"ID":"552589d3-45c7-4814-82c4-2a4d8e7d5718","Type":"ContainerDied","Data":"f60cc303d446c5a91ad5bf113a3a2a2b0455e0943e24ed49f46f81c7d8bfff03"}
May 25 11:38:32.015766 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.015624    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"
May 25 11:38:32.015766 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.015637    2556 scope.go:117] "RemoveContainer" containerID="3ab8db91d8989512f18aea62927b88e4ce745546c701f5955f5a2f461d613479"
May 25 11:38:32.017612 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.017547    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5fplw/must-gather-8mbz5" event={"ID":"5448ccbb-988b-49a5-8a85-980ad8e428a0","Type":"ContainerStarted","Data":"8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51"}
May 25 11:38:32.017612 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.017582    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5fplw/must-gather-8mbz5" event={"ID":"5448ccbb-988b-49a5-8a85-980ad8e428a0","Type":"ContainerStarted","Data":"42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6"}
May 25 11:38:32.029669 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.023806    2556 scope.go:117] "RemoveContainer" containerID="d545d4ef2346d7a8392d161b541efd923fc0f131b1b54c9eccaf1324e7be3268"
May 25 11:38:32.034005 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.033980    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"]
May 25 11:38:32.038420 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.038396    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-1bc1a-predictor-c9bc6dd95-pm2rc"]
May 25 11:38:32.039044 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.039026    2556 scope.go:117] "RemoveContainer" containerID="02d10bac5a5b466e930a3d4d5a6e6588dc4c24d90a03c78ee24269f5c1e64582"
May 25 11:38:32.053381 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:32.053339    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-5fplw/must-gather-8mbz5" podStartSLOduration=0.956477716 podStartE2EDuration="7.053327518s" podCreationTimestamp="2026-05-25 11:38:25 +0000 UTC" firstStartedPulling="2026-05-25 11:38:25.623163168 +0000 UTC m=+1254.253323960" lastFinishedPulling="2026-05-25 11:38:31.720012975 +0000 UTC m=+1260.350173762" observedRunningTime="2026-05-25 11:38:32.051436916 +0000 UTC m=+1260.681597726" watchObservedRunningTime="2026-05-25 11:38:32.053327518 +0000 UTC m=+1260.683488327"
May 25 11:38:33.981043 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:33.981010    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" path="/var/lib/kubelet/pods/552589d3-45c7-4814-82c4-2a4d8e7d5718/volumes"
May 25 11:38:51.080916 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:51.080887    2556 generic.go:358] "Generic (PLEG): container finished" podID="5448ccbb-988b-49a5-8a85-980ad8e428a0" containerID="42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6" exitCode=0
May 25 11:38:51.081244 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:51.080958    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5fplw/must-gather-8mbz5" event={"ID":"5448ccbb-988b-49a5-8a85-980ad8e428a0","Type":"ContainerDied","Data":"42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6"}
May 25 11:38:51.081244 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:51.081236    2556 scope.go:117] "RemoveContainer" containerID="42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6"
May 25 11:38:52.016004 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.015973    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5fplw_must-gather-8mbz5_5448ccbb-988b-49a5-8a85-980ad8e428a0/gather/0.log"
May 25 11:38:52.587955 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.587906    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mqk4j/must-gather-dfhws"]
May 25 11:38:52.588427 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.588408    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kube-rbac-proxy"
May 25 11:38:52.588522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.588431    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kube-rbac-proxy"
May 25 11:38:52.588522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.588449    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container"
May 25 11:38:52.588522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.588458    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container"
May 25 11:38:52.588522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.588472    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="storage-initializer"
May 25 11:38:52.588522 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.588499    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="storage-initializer"
May 25 11:38:52.588783 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.588603    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kserve-container"
May 25 11:38:52.588783 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.588616    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="552589d3-45c7-4814-82c4-2a4d8e7d5718" containerName="kube-rbac-proxy"
May 25 11:38:52.590706 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.590686    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mqk4j/must-gather-dfhws"
May 25 11:38:52.593905 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.593887    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mqk4j\"/\"kube-root-ca.crt\""
May 25 11:38:52.594028 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.594016    2556 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mqk4j\"/\"openshift-service-ca.crt\""
May 25 11:38:52.594098 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.594018    2556 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mqk4j\"/\"default-dockercfg-s2zl5\""
May 25 11:38:52.603929 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.603908    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mqk4j/must-gather-dfhws"]
May 25 11:38:52.701901 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.701879    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8397313f-ab51-419e-bd9e-8f2e4858af7a-must-gather-output\") pod \"must-gather-dfhws\" (UID: \"8397313f-ab51-419e-bd9e-8f2e4858af7a\") " pod="openshift-must-gather-mqk4j/must-gather-dfhws"
May 25 11:38:52.702015 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.701918    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rpsp\" (UniqueName: \"kubernetes.io/projected/8397313f-ab51-419e-bd9e-8f2e4858af7a-kube-api-access-5rpsp\") pod \"must-gather-dfhws\" (UID: \"8397313f-ab51-419e-bd9e-8f2e4858af7a\") " pod="openshift-must-gather-mqk4j/must-gather-dfhws"
May 25 11:38:52.802670 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.802647    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8397313f-ab51-419e-bd9e-8f2e4858af7a-must-gather-output\") pod \"must-gather-dfhws\" (UID: \"8397313f-ab51-419e-bd9e-8f2e4858af7a\") " pod="openshift-must-gather-mqk4j/must-gather-dfhws"
May 25 11:38:52.802770 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.802682    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5rpsp\" (UniqueName: \"kubernetes.io/projected/8397313f-ab51-419e-bd9e-8f2e4858af7a-kube-api-access-5rpsp\") pod \"must-gather-dfhws\" (UID: \"8397313f-ab51-419e-bd9e-8f2e4858af7a\") " pod="openshift-must-gather-mqk4j/must-gather-dfhws"
May 25 11:38:52.802975 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.802957    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8397313f-ab51-419e-bd9e-8f2e4858af7a-must-gather-output\") pod \"must-gather-dfhws\" (UID: \"8397313f-ab51-419e-bd9e-8f2e4858af7a\") " pod="openshift-must-gather-mqk4j/must-gather-dfhws"
May 25 11:38:52.812092 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.812065    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rpsp\" (UniqueName: \"kubernetes.io/projected/8397313f-ab51-419e-bd9e-8f2e4858af7a-kube-api-access-5rpsp\") pod \"must-gather-dfhws\" (UID: \"8397313f-ab51-419e-bd9e-8f2e4858af7a\") " pod="openshift-must-gather-mqk4j/must-gather-dfhws"
May 25 11:38:52.899349 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:52.899296    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mqk4j/must-gather-dfhws"
May 25 11:38:53.020899 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:53.020816    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mqk4j/must-gather-dfhws"]
May 25 11:38:53.023129 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:38:53.023102    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8397313f_ab51_419e_bd9e_8f2e4858af7a.slice/crio-f82a3f46a1d565f07866343d3bd639fad4b53163627f9a6ae83a0bf4862e2b26 WatchSource:0}: Error finding container f82a3f46a1d565f07866343d3bd639fad4b53163627f9a6ae83a0bf4862e2b26: Status 404 returned error can't find the container with id f82a3f46a1d565f07866343d3bd639fad4b53163627f9a6ae83a0bf4862e2b26
May 25 11:38:53.087396 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:53.087363    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mqk4j/must-gather-dfhws" event={"ID":"8397313f-ab51-419e-bd9e-8f2e4858af7a","Type":"ContainerStarted","Data":"f82a3f46a1d565f07866343d3bd639fad4b53163627f9a6ae83a0bf4862e2b26"}
May 25 11:38:54.092559 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:54.092526    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mqk4j/must-gather-dfhws" event={"ID":"8397313f-ab51-419e-bd9e-8f2e4858af7a","Type":"ContainerStarted","Data":"80c9fd8c8cd137cd695659cafdc37f6e0832091f6c536da5bdffc4988027b4d9"}
May 25 11:38:55.099514 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:55.099452    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mqk4j/must-gather-dfhws" event={"ID":"8397313f-ab51-419e-bd9e-8f2e4858af7a","Type":"ContainerStarted","Data":"b7806bb089312d783530f62f137fe4c68f9222ef1fe853503bb30db3eab23045"}
May 25 11:38:55.115881 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:55.115827    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-mqk4j/must-gather-dfhws" podStartSLOduration=2.152187576 podStartE2EDuration="3.115809972s" podCreationTimestamp="2026-05-25 11:38:52 +0000 UTC" firstStartedPulling="2026-05-25 11:38:53.024960547 +0000 UTC m=+1281.655121339" lastFinishedPulling="2026-05-25 11:38:53.988582947 +0000 UTC m=+1282.618743735" observedRunningTime="2026-05-25 11:38:55.114949595 +0000 UTC m=+1283.745110406" watchObservedRunningTime="2026-05-25 11:38:55.115809972 +0000 UTC m=+1283.745970783"
May 25 11:38:55.391046 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:55.390967    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m5jhd_08a5630b-4d2a-493b-84d6-56f03d88d747/global-pull-secret-syncer/0.log"
May 25 11:38:55.568613 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:55.568588    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n2d2h_fb296f1f-d101-4bd7-a981-3803bc01c51a/konnectivity-agent/0.log"
May 25 11:38:55.620471 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:55.620441    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-161.ec2.internal_b3204d4e276a6036a6def62b77dcf7af/haproxy/0.log"
May 25 11:38:57.419893 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.419852    2556 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-5fplw/must-gather-8mbz5"]
May 25 11:38:57.420701 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.420667    2556 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-5fplw/must-gather-8mbz5" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" containerName="copy" containerID="cri-o://8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51" gracePeriod=2
May 25 11:38:57.425170 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.425139    2556 status_manager.go:895] "Failed to get status for pod" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" pod="openshift-must-gather-5fplw/must-gather-8mbz5" err="pods \"must-gather-8mbz5\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5fplw\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object"
May 25 11:38:57.427145 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.427114    2556 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-5fplw/must-gather-8mbz5"]
May 25 11:38:57.776247 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.776206    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5fplw_must-gather-8mbz5_5448ccbb-988b-49a5-8a85-980ad8e428a0/copy/0.log"
May 25 11:38:57.779517 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.776634    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:57.785501 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.782016    2556 status_manager.go:895] "Failed to get status for pod" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" pod="openshift-must-gather-5fplw/must-gather-8mbz5" err="pods \"must-gather-8mbz5\" is forbidden: User \"system:node:ip-10-0-128-161.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-5fplw\": no relationship found between node 'ip-10-0-128-161.ec2.internal' and this object"
May 25 11:38:57.853028 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.852995    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5448ccbb-988b-49a5-8a85-980ad8e428a0-must-gather-output\") pod \"5448ccbb-988b-49a5-8a85-980ad8e428a0\" (UID: \"5448ccbb-988b-49a5-8a85-980ad8e428a0\") "
May 25 11:38:57.856231 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.855575    2556 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t8kd\" (UniqueName: \"kubernetes.io/projected/5448ccbb-988b-49a5-8a85-980ad8e428a0-kube-api-access-5t8kd\") pod \"5448ccbb-988b-49a5-8a85-980ad8e428a0\" (UID: \"5448ccbb-988b-49a5-8a85-980ad8e428a0\") "
May 25 11:38:57.856452 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.854611    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5448ccbb-988b-49a5-8a85-980ad8e428a0-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "5448ccbb-988b-49a5-8a85-980ad8e428a0" (UID: "5448ccbb-988b-49a5-8a85-980ad8e428a0"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
May 25 11:38:57.869608 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.869563    2556 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5448ccbb-988b-49a5-8a85-980ad8e428a0-kube-api-access-5t8kd" (OuterVolumeSpecName: "kube-api-access-5t8kd") pod "5448ccbb-988b-49a5-8a85-980ad8e428a0" (UID: "5448ccbb-988b-49a5-8a85-980ad8e428a0"). InnerVolumeSpecName "kube-api-access-5t8kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
May 25 11:38:57.956531 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.956471    2556 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5448ccbb-988b-49a5-8a85-980ad8e428a0-must-gather-output\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:38:57.956531 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.956532    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t8kd\" (UniqueName: \"kubernetes.io/projected/5448ccbb-988b-49a5-8a85-980ad8e428a0-kube-api-access-5t8kd\") on node \"ip-10-0-128-161.ec2.internal\" DevicePath \"\""
May 25 11:38:57.986682 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:57.983926    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" path="/var/lib/kubelet/pods/5448ccbb-988b-49a5-8a85-980ad8e428a0/volumes"
May 25 11:38:58.114451 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:58.113428    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-5fplw_must-gather-8mbz5_5448ccbb-988b-49a5-8a85-980ad8e428a0/copy/0.log"
May 25 11:38:58.114451 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:58.113806    2556 generic.go:358] "Generic (PLEG): container finished" podID="5448ccbb-988b-49a5-8a85-980ad8e428a0" containerID="8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51" exitCode=143
May 25 11:38:58.114451 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:58.113870    2556 scope.go:117] "RemoveContainer" containerID="8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51"
May 25 11:38:58.114451 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:58.114009    2556 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5fplw/must-gather-8mbz5"
May 25 11:38:58.133789 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:58.133603    2556 scope.go:117] "RemoveContainer" containerID="42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6"
May 25 11:38:58.155238 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:58.154477    2556 scope.go:117] "RemoveContainer" containerID="8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51"
May 25 11:38:58.155238 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:38:58.154828    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51\": container with ID starting with 8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51 not found: ID does not exist" containerID="8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51"
May 25 11:38:58.155238 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:58.154862    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51"} err="failed to get container status \"8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51\": rpc error: code = NotFound desc = could not find container \"8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51\": container with ID starting with 8d76a146559dc971ea0817537b5714748a7505ec50550291b20f6fa99059fb51 not found: ID does not exist"
May 25 11:38:58.155238 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:58.154889    2556 scope.go:117] "RemoveContainer" containerID="42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6"
May 25 11:38:58.155238 ip-10-0-128-161 kubenswrapper[2556]: E0525 11:38:58.155147    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6\": container with ID starting with 42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6 not found: ID does not exist" containerID="42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6"
May 25 11:38:58.155238 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:58.155172    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6"} err="failed to get container status \"42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6\": rpc error: code = NotFound desc = could not find container \"42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6\": container with ID starting with 42c4dc7c44746fb16900737c441d0e8c6fb09bfc9a6f322ad13ff6537b6e89b6 not found: ID does not exist"
May 25 11:38:59.081575 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.081497    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea/alertmanager/0.log"
May 25 11:38:59.107999 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.107968    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea/config-reloader/0.log"
May 25 11:38:59.136831 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.136778    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea/kube-rbac-proxy-web/0.log"
May 25 11:38:59.159300 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.159249    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea/kube-rbac-proxy/0.log"
May 25 11:38:59.183162 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.183124    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea/kube-rbac-proxy-metric/0.log"
May 25 11:38:59.205867 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.205837    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea/prom-label-proxy/0.log"
May 25 11:38:59.233434 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.233405    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f7bf0a0c-b3df-4d4f-aefb-20728d6f37ea/init-config-reloader/0.log"
May 25 11:38:59.311304 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.311266    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-klctg_8ab2afe6-3d4c-4562-892d-e5bfa9a67071/kube-state-metrics/0.log"
May 25 11:38:59.332255 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.332138    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-klctg_8ab2afe6-3d4c-4562-892d-e5bfa9a67071/kube-rbac-proxy-main/0.log"
May 25 11:38:59.358857 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.358775    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-klctg_8ab2afe6-3d4c-4562-892d-e5bfa9a67071/kube-rbac-proxy-self/0.log"
May 25 11:38:59.398361 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.398329    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6cd8f9d848-f46g9_0277fb40-a9db-499f-aeb8-2e3ff321f967/metrics-server/0.log"
May 25 11:38:59.424971 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.424935    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-5cd8r_b2772063-b043-4d74-996e-14736d635b27/monitoring-plugin/0.log"
May 25 11:38:59.630994 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.630891    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zk687_5d4d1611-41f7-4e75-a5a0-de5d58e26b83/node-exporter/0.log"
May 25 11:38:59.660836 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.660816    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zk687_5d4d1611-41f7-4e75-a5a0-de5d58e26b83/kube-rbac-proxy/0.log"
May 25 11:38:59.697010 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.696984    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zk687_5d4d1611-41f7-4e75-a5a0-de5d58e26b83/init-textfile/0.log"
May 25 11:38:59.729744 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.729716    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-gnt7w_90c1a143-cca8-4994-8e03-3d76eb3bb115/kube-rbac-proxy-main/0.log"
May 25 11:38:59.754726 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.754683    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-gnt7w_90c1a143-cca8-4994-8e03-3d76eb3bb115/kube-rbac-proxy-self/0.log"
May 25 11:38:59.777601 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.777561    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-gnt7w_90c1a143-cca8-4994-8e03-3d76eb3bb115/openshift-state-metrics/0.log"
May 25 11:38:59.815545 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.815512    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2270ae61-5ab6-41f2-8203-77e458a8d22a/prometheus/0.log"
May 25 11:38:59.839377 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.839345    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2270ae61-5ab6-41f2-8203-77e458a8d22a/config-reloader/0.log"
May 25 11:38:59.867506 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.867465    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2270ae61-5ab6-41f2-8203-77e458a8d22a/thanos-sidecar/0.log"
May 25 11:38:59.898218 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.898105    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2270ae61-5ab6-41f2-8203-77e458a8d22a/kube-rbac-proxy-web/0.log"
May 25 11:38:59.924914 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.924885    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2270ae61-5ab6-41f2-8203-77e458a8d22a/kube-rbac-proxy/0.log"
May 25 11:38:59.946772 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.946738    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2270ae61-5ab6-41f2-8203-77e458a8d22a/kube-rbac-proxy-thanos/0.log"
May 25 11:38:59.969570 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:38:59.969537    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_2270ae61-5ab6-41f2-8203-77e458a8d22a/init-config-reloader/0.log"
May 25 11:39:00.001258 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.001223    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-57wdg_b9bd5baa-a1c8-446d-b688-e707d041b40c/prometheus-operator/0.log"
May 25 11:39:00.023666 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.023633    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-57wdg_b9bd5baa-a1c8-446d-b688-e707d041b40c/kube-rbac-proxy/0.log"
May 25 11:39:00.051320 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.051285    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-s8wf5_3b3de746-239e-44c7-94bd-60d0a466b736/prometheus-operator-admission-webhook/0.log"
May 25 11:39:00.082370 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.082346    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5ffd9b9b68-wsxpv_85b0e62e-f512-492b-8e96-145bf0452b55/telemeter-client/0.log"
May 25 11:39:00.107197 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.107170    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5ffd9b9b68-wsxpv_85b0e62e-f512-492b-8e96-145bf0452b55/reload/0.log"
May 25 11:39:00.152648 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.152569    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5ffd9b9b68-wsxpv_85b0e62e-f512-492b-8e96-145bf0452b55/kube-rbac-proxy/0.log"
May 25 11:39:00.184101 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.184075    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b7cd5f99d-68bll_c9a65038-2eab-4247-9284-b3ff2a36b89c/thanos-query/0.log"
May 25 11:39:00.208462 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.208432    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b7cd5f99d-68bll_c9a65038-2eab-4247-9284-b3ff2a36b89c/kube-rbac-proxy-web/0.log"
May 25 11:39:00.231329 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.231293    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b7cd5f99d-68bll_c9a65038-2eab-4247-9284-b3ff2a36b89c/kube-rbac-proxy/0.log"
May 25 11:39:00.263958 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.263931    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b7cd5f99d-68bll_c9a65038-2eab-4247-9284-b3ff2a36b89c/prom-label-proxy/0.log"
May 25 11:39:00.297629 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.297600    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b7cd5f99d-68bll_c9a65038-2eab-4247-9284-b3ff2a36b89c/kube-rbac-proxy-rules/0.log"
May 25 11:39:00.322272 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:00.322241    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6b7cd5f99d-68bll_c9a65038-2eab-4247-9284-b3ff2a36b89c/kube-rbac-proxy-metrics/0.log"
May 25 11:39:02.486892 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.486861    2556 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"]
May 25 11:39:02.487348 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.487199    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" containerName="gather"
May 25 11:39:02.487348 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.487210    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" containerName="gather"
May 25 11:39:02.487348 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.487225    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" containerName="copy"
May 25 11:39:02.487348 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.487232    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" containerName="copy"
May 25 11:39:02.487348 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.487288    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" containerName="copy"
May 25 11:39:02.487348 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.487296    2556 memory_manager.go:356] "RemoveStaleState removing state" podUID="5448ccbb-988b-49a5-8a85-980ad8e428a0" containerName="gather"
May 25 11:39:02.490652 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.490631    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.501756 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.501731    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"]
May 25 11:39:02.598158 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.598112    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f2b4\" (UniqueName: \"kubernetes.io/projected/e8fccff1-6ac5-4e50-884c-494d9b579d93-kube-api-access-5f2b4\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.598341 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.598185    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-lib-modules\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.598341 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.598224    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-proc\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.598341 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.598247    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-podres\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.598341 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.598306    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-sys\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.699385 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.699351    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-proc\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.699563 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.699395    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-podres\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.699563 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.699447    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-sys\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.699563 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.699455    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-proc\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.699563 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.699537    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-sys\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.699740 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.699574    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5f2b4\" (UniqueName: \"kubernetes.io/projected/e8fccff1-6ac5-4e50-884c-494d9b579d93-kube-api-access-5f2b4\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.699740 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.699597    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-podres\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.699740 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.699604    2556 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-lib-modules\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.699740 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.699687    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e8fccff1-6ac5-4e50-884c-494d9b579d93-lib-modules\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.710568 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.710538    2556 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f2b4\" (UniqueName: \"kubernetes.io/projected/e8fccff1-6ac5-4e50-884c-494d9b579d93-kube-api-access-5f2b4\") pod \"perf-node-gather-daemonset-4b9kk\" (UID: \"e8fccff1-6ac5-4e50-884c-494d9b579d93\") " pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.801737 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.801659    2556 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:02.951255 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:02.951208    2556 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"]
May 25 11:39:02.954432 ip-10-0-128-161 kubenswrapper[2556]: W0525 11:39:02.954402    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode8fccff1_6ac5_4e50_884c_494d9b579d93.slice/crio-54fed8a12fe61506db0ee58e9af10e1af6be92660c54c80c4e5cb0221c95a6b7 WatchSource:0}: Error finding container 54fed8a12fe61506db0ee58e9af10e1af6be92660c54c80c4e5cb0221c95a6b7: Status 404 returned error can't find the container with id 54fed8a12fe61506db0ee58e9af10e1af6be92660c54c80c4e5cb0221c95a6b7
May 25 11:39:03.133691 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:03.133659    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk" event={"ID":"e8fccff1-6ac5-4e50-884c-494d9b579d93","Type":"ContainerStarted","Data":"6fd08a71b477aab87de55d7eb5c4d8399ecc8c1dbb866b55441a93177b62c33f"}
May 25 11:39:03.133691 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:03.133694    2556 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk" event={"ID":"e8fccff1-6ac5-4e50-884c-494d9b579d93","Type":"ContainerStarted","Data":"54fed8a12fe61506db0ee58e9af10e1af6be92660c54c80c4e5cb0221c95a6b7"}
May 25 11:39:03.133868 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:03.133802    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:03.152628 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:03.152586    2556 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk" podStartSLOduration=1.152569624 podStartE2EDuration="1.152569624s" podCreationTimestamp="2026-05-25 11:39:02 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-25 11:39:03.15065345 +0000 UTC m=+1291.780814273" watchObservedRunningTime="2026-05-25 11:39:03.152569624 +0000 UTC m=+1291.782730433"
May 25 11:39:03.358243 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:03.358163    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pw56p_7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4/dns/0.log"
May 25 11:39:03.379201 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:03.379179    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pw56p_7e6f9fdb-3081-4d28-b6e5-d2d706f7aeb4/kube-rbac-proxy/0.log"
May 25 11:39:03.526079 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:03.526056    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hp2v9_aec6f6e8-33f0-43d3-b580-0b98f54b3bf3/dns-node-resolver/0.log"
May 25 11:39:04.028930 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:04.028904    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v2shg_fdef210b-a23e-438e-a7d9-7978931d243e/node-ca/0.log"
May 25 11:39:05.052681 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:05.052654    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fgzh7_2f5aae4f-4930-47cc-be9f-2b971dbd12af/serve-healthcheck-canary/0.log"
May 25 11:39:05.422787 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:05.422715    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggmkh_32313dcb-927f-4c70-917c-f876bfc59823/kube-rbac-proxy/0.log"
May 25 11:39:05.444117 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:05.444092    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggmkh_32313dcb-927f-4c70-917c-f876bfc59823/exporter/0.log"
May 25 11:39:05.465414 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:05.465390    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ggmkh_32313dcb-927f-4c70-917c-f876bfc59823/extractor/0.log"
May 25 11:39:07.710047 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:07.710015    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-696fc77849-c9c6b_fd326d61-80e0-44e8-be8b-ff6c77610195/manager/0.log"
May 25 11:39:07.729950 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:07.729927    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-n26fb_5e991ab6-5bc2-44fc-b170-841217759daf/s3-init/0.log"
May 25 11:39:09.145670 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:09.145645    2556 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mqk4j/perf-node-gather-daemonset-4b9kk"
May 25 11:39:13.399616 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.399590    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sh48r_13fe7228-2e31-41ea-866c-b1c045e78460/kube-multus-additional-cni-plugins/0.log"
May 25 11:39:13.424455 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.424433    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sh48r_13fe7228-2e31-41ea-866c-b1c045e78460/egress-router-binary-copy/0.log"
May 25 11:39:13.445352 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.445329    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sh48r_13fe7228-2e31-41ea-866c-b1c045e78460/cni-plugins/0.log"
May 25 11:39:13.466519 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.466477    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sh48r_13fe7228-2e31-41ea-866c-b1c045e78460/bond-cni-plugin/0.log"
May 25 11:39:13.488320 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.488299    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sh48r_13fe7228-2e31-41ea-866c-b1c045e78460/routeoverride-cni/0.log"
May 25 11:39:13.509494 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.509459    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sh48r_13fe7228-2e31-41ea-866c-b1c045e78460/whereabouts-cni-bincopy/0.log"
May 25 11:39:13.530622 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.530560    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sh48r_13fe7228-2e31-41ea-866c-b1c045e78460/whereabouts-cni/0.log"
May 25 11:39:13.595666 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.595636    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-hkkfj_bb98fe05-e021-483f-baf0-1e56cc1ad554/kube-multus/0.log"
May 25 11:39:13.649712 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.649641    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8w9qg_37262c21-a42a-4ff7-90c7-bbe50a05d68b/network-metrics-daemon/0.log"
May 25 11:39:13.670899 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:13.670874    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8w9qg_37262c21-a42a-4ff7-90c7-bbe50a05d68b/kube-rbac-proxy/0.log"
May 25 11:39:14.474826 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:14.474800    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-controller/0.log"
May 25 11:39:14.493940 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:14.493913    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/0.log"
May 25 11:39:14.500725 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:14.500686    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovn-acl-logging/1.log"
May 25 11:39:14.517241 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:14.517223    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/kube-rbac-proxy-node/0.log"
May 25 11:39:14.537833 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:14.537816    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/kube-rbac-proxy-ovn-metrics/0.log"
May 25 11:39:14.557311 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:14.557291    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/northd/0.log"
May 25 11:39:14.576262 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:14.576245    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/nbdb/0.log"
May 25 11:39:14.602456 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:14.602442    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/sbdb/0.log"
May 25 11:39:14.711313 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:14.711291    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-794p7_9607124d-82db-4c28-a3a4-5a49d4603955/ovnkube-controller/0.log"
May 25 11:39:16.361455 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:16.361426    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2j6dr_4ed96c55-4683-4498-94f8-a8865cd52373/network-check-target-container/0.log"
May 25 11:39:17.425451 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:17.425427    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-879wd_9d72f58b-2d2b-49e1-9d22-42b6fa8d8f90/iptables-alerter/0.log"
May 25 11:39:18.157705 ip-10-0-128-161 kubenswrapper[2556]: I0525 11:39:18.157680    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-9sv9g_af6ff664-f55f-4650-b65a-3816ace0d8c0/tuned/0.log"